CHANGES 21.5 KB
Newer Older
Per Hedbor's avatar
Per Hedbor committed
1
Pike 8.1: Changes since Pike 8.0 (scratch area for future release notes)
2
------------------------------------------------------------------------
3

Martin Nilsson's avatar
Martin Nilsson committed
4 5 6 7 8 9 10 11
New language features
---------------------

o Unlimited character constant size.

  There is no longer any limit to the size of character constants,
  e.g. 'acdefghijk' creates the bignum 0x61636465666768696a6b.

Per Hedbor's avatar
Per Hedbor committed
12 13 14 15 16
o 'auto' type added.

  This is much like a strict typed mixed. The actual type is deduced
  compile-time.

17 18 19
  Typical use is as the type for variables in foreach when looping over
  complexly typed values, or as the type in macro-defined functions and
  similar.
Per Hedbor's avatar
Per Hedbor committed
20 21

  auto can be used as return type for a function, it will be the
22
  type-union of all the types of the return statements in the
Per Hedbor's avatar
Per Hedbor committed
23 24 25 26 27 28 29 30 31 32 33 34
  function.

o typeof(X) can now be used as a type.

  The main use case is in macros, but it might be useful for other
  things as well (as an example in a typedef).

  typedef typeof(Val.true)|typeof(Val.false) bool;

  typeof, when used like this, behaves very much like the C++
  decltype() expression.

35
o Shorthand for int(val..val) added: int(val) .
36 37 38 39

  A syntax for when the range of an integer type only
  contains a single value.

40
o Shorthand for string(val..val) added: string(val) .
41 42 43 44

  A syntax for when the range of a string type only
  contains a single value.

45 46 47 48
o Syntax for specifying the length of array and string types.

  Eg string(4bit: 8bit) is 8-bit strings of lengths 0-15.

Per Hedbor's avatar
Per Hedbor committed
49 50 51 52 53 54 55 56
o ** operator added. It's exponentiation and works with most
  combination of numerical types (int,float,Gmp.mpq,Gmp.mpf,Gmp.mpz)

  `** and ``** operator overloading functions added.

  This introduces one incompatible change: Previously the "pow"
  function called a "_pow" function in the first argument if it was an
  object. It has now been changed to also use `** (or, rather, pow()
57
  is now implemented using predef::`**()).
Per Hedbor's avatar
Per Hedbor committed
58

59 60 61
* Binary or between a multiset and UNDEFINED is now behaves like
  binary or between the multiset and the empty multiset.

62 63 64 65 66 67
o Three pass compiler.

  An extra pass of the compiler has been added. It is used to properly
  resolve types in forward referring expressions, and is only run when
  needed.

68 69
o #pragma disassemble

70 71
  This directive is now available even in pikes compiled --without-debug.

72 73 74 75 76
  The byte code output is now interleaved with the generated machine code
  on selected architectures.

o Complain about redundant backslash escapes.

77 78 79
o '__weak__' modifier added.

  It is now possible to declare object variables to have weak references.
80

81 82 83 84 85
o Support for arguments to implicit lambdas added.

  Implicit lambdas now behave as if they were declared as
  lambda(mixed ... __ARGS__).

86 87
  Support for this feature can be detected with #ifdef __ARGS__.

88 89 90 91
o Function local function declarations are now statements
  (and not expressions). This means that there is no longer
  any need to terminate them with a semicolon.

92 93 94 95 96 97
o Anonymous class definitions are now always expressions.

o Function local class definitions are now statements
  (and not expressions). This means that there is no longer
  any need to terminate them with a semicolon.

98 99 100 101 102
o The inherit statement's argument is now a proper
  constant expression. This means that it is possible
  to dynamically (at compile-time) generate the value
  to be inherited, eg with Pike.ProxyFactory().

103 104
o Complain about shadowed variant functions.

105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122
o Syntax for generator functions added.

  continue is now treated as a modifier, and causes such functions
  to become generator functions, where calling the function sets
  up a restartable function context.

  continue may also preceed return statements, which causes
  the generator function state to be saved before returning.

  Alternatively the "function" Pike.yield() may be used to return
  a value, and restart with a value from the restarter.

  Restartable functions get the type signature

    X fun(mixed|void value, function(mixed|void: void)|void resume_cb)

  where X is the declared return type of the generator function,
  value is the value to be returned by Pike.yield(), and resume_cb
123 124
  is a function that will be called at the yield point. resume_cb
  may be used to eg behave as if Pike.yield() had thrown an error.
125

126 127 128 129 130 131
o Machine code support for more architectures.

  There's now machine code support for arm32 and arm64.

o Fixed multiple integer over- and underflow bugs.

132 133 134 135 136 137 138
o Extended sscanf %O

  sscanf() is now able to parse all base types that sprintf %O can
  output. It can now also parse most literal constants pike accepts,
  including integers prefixed with 0x and 0b, and floating point
  numbers in scientific notation.

139 140 141 142 143 144 145
o Returning void

  It is now possible to return void responses from void functions.

    void a() { return; }
    void b() { return a(); }

146

147 148 149
Bug fixes
---------

150 151
o Compiler

152 153 154 155 156 157 158 159 160
  - Constant integers outside the 32-bit signed range not longer
    get their types mapped to int(-0x80000000) or int(0x7fffffff),
    (depending on sign) but instead keep the generic int type.
    This fixes issues where the type resolution system derived
    erroneous types from the truncated types.

  - Under some circumstances the compiler would insert erroneous
    F_ESCAPE_CATCH opcodes. This could cause very strange
    runtime errors.
161

162 163 164
  - The compiler could get confused by combinations of labels
    with inline functions or constant expressions.

165 166 167 168 169
o Runtime

  The runtime could get confused by PROGRAM_DESTRUCT_IMMEDIATE
  objects having destruct callbacks under some circumstances.

170 171 172 173 174 175
o Operator functions

  Calling operator functions with more than two arguments will now
  work correctly for objects, where previously only the first two
  objects where added.

176 177 178 179
  When adding arrays, multisets and mappings, UNDEFINED will now
  always be ignored. Previously it was only ignored when first in the
  argument list, otherwise an exception would be thrown.

180 181 182 183 184
  The LFUNs will now be called with a consistent number of
  arguments. Pike implementations rarely implemented varargs on
  operator LFUNs, so this change should address many potential hidden
  errors.

185 186 187 188 189 190 191
o Pike.Backend

  Backend callbacks throwing errors could cause backends to
  enter a state where they got stuck throwing "Backend already
  running - cannot reenter.". This was typically triggered via
  the Stdio.sendfile() result callback.

192 193 194
o Protocols.HTTP.Server

  The server module will now read payloads for HTTP PUT requests, just
195 196 197 198 199 200 201 202 203 204 205 206 207 208
  as any other method. Previously it would stop reading the body and
  it was up to the caller to read enough data from the socket and
  combine with already read data.

  Setting "connection" header in the "extra_heads" to Request object
  method response_and_finish() will now control if keep-alive should be
  used or not. Otherwise it will be decided by the clients request
  headers, as previously.

  The headers "content-type", "content-range", "content-length",
  "server", "date" and "last-modified" will not be added or
  overwritten if provided in the "extra_heads".

  Header names in "extra_heads" will not have their case modified.
209

210 211 212 213
o Standards.JSON and Standards.JSON5

  encode() now allows other threads to run every now and then.

214 215 216
o ADT.OrderedMapping

  Adding duplicate keys now replaces the existing keys.
217

218

219 220 221
Incompatible changes
--------------------

222 223 224 225 226 227
o Sql.Sql is no longer a wrapper class, but a function.

  The wrapper class has been obsoleted by introduction of the
  new base class __builtin.Sql.Connection. Note that it is
  still possible to use Sql.Sql as the type of connection objects.

228 229 230 231
o Stdio.Port now has a default id of itself (as documented).

  Previously the default id for Stdio.Port objects was 0.

232 233 234 235 236 237 238 239 240 241 242
o Named classes are always statements.

  As mentioned above, it is no longer possible to define a named
  class in an expression. Such code has been observed, but none
  where the name was actually used. Use an anonymous class instead.

o Anonymous classes are always expressions.

  It used to be possible (but typically not useful) to define
  an anonymous class as a statement. Add a name for the class.

243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269
o Gz.crc32 and Nettle.crc32c now only return positive results.

o glob() has changed.

  The main incompatibilities are that [ and \ are now special
  characters in the pattern, and if you check the return value against
  1 instead of checking if it's true, and use an array as the first
  argument you will have to change your code.

  This is in order to extend the glob function to cover what
  'standard' glob functions do:

  glob() now accepts quotes (\* to match a single *, as an example)
  and handles ranges ([abc] for a, b or c, [a-z0-9] for a single
  character between a and z or 0-9

  You can also negate a range using ^ or ! ([^a-zA-Z] as an example).

  When the first argument (the pattern) to glob is an array, glob now
  returns which pattern in the array matched.

o hash() has been changed to use siphash. The old hash function is
  available under the name hash_8_0(). Note that this is not a
  cryptographic hash function.

o Stdio.UDP()->send() no longer throws errors on EMSGSIZE and EWOULDBLOCK.

270 271
o When a thread exits by throwing, the thrown value is propagated to
  wait(). Previously it was sent directly to master()->handle_error().
272 273 274

o String.trim_all_whites() renamed String.trim().

275 276 277
o Floats are no longer by default sorted before added. This may reduce
  the precision but increases the speed of adding large number of
  floats by a factor of 500. Applications handling floats with large
Chris Angelico's avatar
Chris Angelico committed
278
  differences in magnitude need to apply the appropriate sorting
279 280 281
  before arithmetics. As `+ was the only operator that performed
  sorting, and functions like Array.sum did not, this was already a
  concern.
282

283 284 285
o Returning UNDEFINED from `+ and ``+ is not allowed and will cause an
  exception.

286 287 288
o RegGetValue(), RegGetKeyNames(), RegGetKeyValues(), openlog(),
  syslog() and closelog() have been moved to System.

289

290 291 292 293 294 295
New modules and functions
-------------------------

o predef::m_add()

  This function adds (as opposed to inserts) an element to a multiset.
296

297 298 299 300 301
o ADT.Scheduler & ADT.TreeScheduler

  These are variants of ADT.Heap where elements typically aren't
  removed from the heap.

302
o Apple.Keychain
303 304 305 306

  Parser for Apple Keychain format files (like the ones in
  /System/Library/Keychains/).

307
o __builtin.Sql
308 309 310

  Generic base classes for Sql API modules.

311 312
o Filesystem.Zip
o Function.bind()
313 314 315 316

  Partially evaluate a function. Similar to Function.curry, but with
  better granularity.

317 318
o Standards.HPack (RFC 7541)
o Stdio.FakePipe
319 320 321

  A simulated Stdio.Pipe.

322 323 324 325 326 327 328 329 330
o Parser.Markdown

o Crypto.Checksum

  This module collect non-cryptographic checksums. Support for crc32,
  adler32 and Castagnoli CRC (CRC32C).

  NB: In the future these may be amended to support the Crypto.Hash API.

331 332 333 334 335
o Parser.ECMAScript

  This module simply provides a token splitter for
  ECMAScript/JavaScript.

336 337 338 339
o Pike.Annotations

  Multiple common annotations are available.

340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359
o Pike.DestructImmediate

  Objects of classes that inherit Pike.DestructImmediate will be
  destructed immediately on their references reaching zero (rather
  than at some arbitrary time "soon"). This class should be used
  for any classes that are (or hold) locks or similar.

o Pike.InhibitDestruct

  Objects of classes that inherit Pike.InhibitDestruct may
  control whether explicit destruct() calls should succeed
  (or be delayed until later). Note that this class is just
  a convenience API, and that this functionality may also
  be implemented by hand. The intended use is for C-classes
  that do not want their storage to be zapped eg during
  library calls.

  Note that it is not possible to inhibit destruction due
  to running out of references or by the gc.

360 361 362 363 364 365 366
o Pike.LiveBacktraceFrame

  This is similar to Pike.BacktraceFrame, but refers two-way
  to an active (or not) execution context. It can be used
  to eg examine (and change) local variables, and is intended
  for debugger use.

367 368 369 370 371 372
o Pike.ProxyFactory()

  This is a function that generates a proxy class for objects
  of the given class.

o Web.Auth & Web.Api
373
o Web.EngineIO & Web.SocketIO
374 375 376 377
o Protocols.HTTP2
o Bittorrent.DHT
o Standards.MsgPack

378 379 380 381
o Thread.RWMutex

  This class implements a multiple reader / single writer mutex. 

382 383 384 385

New features
------------

386 387 388 389 390 391 392
o predef::backtrace()

  predef::backtrace() now takes an optional argument that causes it
  to return LiveBacktraceFrames (instead of BacktraceFrames), which
  may be used to access local variables for the frame while they are
  still in use. This is intended for debugger use.

393 394 395 396
o predef::destruct()

  lfun::_destruct() may now inhibit explicit destruction.

397 398
o predef::equal()

399 400
  equal() on functions now checks if their definitions are the same
  identifier in the same program.
401

402 403 404 405 406
o predef::gc()

  gc() called with a weak mapping as argument now removes weak references
  that are only held by that mapping.

407 408 409 410 411 412 413 414
o predef::m_clear()

  m_clear() now supports operation on multisets and objects.

o predef::m_delete()

  m_delete() now supports operation on multisets.

415 416 417 418 419 420 421
o ADT.Heap

  - An indirection object ADT.Heap.Element has been added to make it
    possible to optimize several heap operations.

  - Added low_pop().

422 423 424 425 426
o Calendar

  - The timezone expert system has been updated for the first time
    in quite a while.

427 428
o Crypto & Nettle

429 430 431
  - Added Curve25519 with EdDSA.

  - Added support for CMAC.
432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459

o Filesystem.Monitor

  The filesystem monitoring system now uses accelleration via
  Inotify et al.

o Gmp

  - mpf is now implemented using gmpf if the library is available.

  - Improved support for GMP 5.0 and later.

o GTK2

  Multiple runtime fixes.

o JOSE (JSON Object Signing and Encryption)

  Some low-level API support has been added to the Crypto and Web
  modules to support parts of RFC 7515 - 7520.

o MasterObject

  - Protect against the same file being compiled concurrently
    in multiple threads.

  - cast_to_program() and cast_to_object() should now be thread safe.

460 461 462 463 464 465 466 467
o CompilerEnvironment()->lock()

  Access to the compiler lock.

o Crypto.ECC.Curve.Point

  A point on an ECC curve.

468 469 470
o Parser.Pike

  Support new language features.
471

472 473 474
o Protocols.DNS

  - Protocols.DNS now supports encoding and decoding CAA RRs.
475

476 477 478
  - Classes derived from Protocols.DNS.server may now override
    report_decode_error() and handle_decode_error() to change how
    errors while decoding a DNS packet are reported and handled.
479 480 481 482 483

o Protocols.WebSocket

  Multiple API changes.

484 485 486
o Random rewrite

  The random functions have been rewritten to ensure security by
487 488 489 490 491
  default. random_string() and random() now get their data directly
  from the operating system random generator, i.e. /dev/urandom on
  most unixen. This is about half the speed compared with the
  random_string function in Pike 8.0, but is now as secure as the
  system random generator.
492 493 494

  For consumers of random data that have additional requirements,
  different random generators are exposed in the new module
495
  Random. The module has the following generators:
496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515

  - Random.Interface

    This is not actually a generator, but an interface class that is
    inherited into all the other generators. It contains code that can
    turn the output from the random_string method into random numbers
    with different limits without introducing bias. It also contains
    code for all the different variants of random() on different
    types. This is currently not possible to implement in Pike code,
    as the typing is too complicated and it is not possible to access
    private/protected _random methods in objects.

  - Random.System

    This generator maps directly on top of the system random
    generator. This is the default generator used for random() and
    random_string().

  - Random.Deterministic

516
    This generator creates the same sequence of random numbers for a
517 518 519 520
    given seed, with good pseudo random properties.

  - Random.Hardware

521
    This generator accesses the hardware random generator, when
522 523 524 525 526
    available.

  - Random.Fast

    This generator takes entropy from the Random.System, but feeds
527
    it into a cryptographic pseudo random number generator to be
528
    able to output data fast. This is not the default random number
529
    generator to avoid loading crypto code on every startup.
530

531 532 533
  Comparing the different generators with each other gives the
  following approximate speeds on a Linux system with hardware
  random support:
534 535 536 537 538 539 540

    Random.System           1.0
    Pike 8.0 random_string  0.45
    Random.Hardware         0.25
    Random.Fast             0.20
    Random.Deterministic    0.20

541 542
  Objects implementing the _random lfun now get two arguments, the
  current random_string() and random() functions. This is convenient
543
  for C-level functions in that they don't need to look up functions
544 545 546 547 548
  themselves. Note that it is possible for a user to replace these
  with non-conforming functions (returning values of the wrong type,
  strings of the wrong length or shift size, and values outside the
  given range) or even non-functions.

549
  All code in Pike that use random numbers now use the current random
550 551 552 553 554 555 556 557 558
  functions (though in some cases fixed at object creation). This
  allows for repeatable results if the random functions are replaced
  with a deterministic random generator, such as
  Random.Deterministic. Example:

    Random.Deterministic rnd = Random.Deterministic( seed );
    add_constant( "random_string", rnd->random_string );
    add_constant( "random", rnd->random );

559
o Sql
560

561
  - Most Sql C-modules converted to cmod.
562

563 564 565 566
  - All Sql driver modules have __builtin.Sql.Connection as
    a common base class.

  - Added next_result() to support queries returning multiple resultsets.
567

568
  - ODBC & tds: Support more datatypes.
Martin Nilsson's avatar
Martin Nilsson committed
569

570
  - ODBC: Support big_typed_query().
Martin Nilsson's avatar
Martin Nilsson committed
571

572
  - pgsql: Lots of changes and fixes.
Per Hedbor's avatar
Per Hedbor committed
573

574 575 576 577
  - pgsql: Toggle cache_autoprepared_statements default to off;
    turning it on triggers a bug in PostgreSQL sometimes
    that causes spikes in CPU usage of the database.

578
o SSL
Per Hedbor's avatar
Per Hedbor committed
579

580
  - Support session tickets.
Per Hedbor's avatar
Per Hedbor committed
581

582
  - Support Negotiated FF-DHE.
Per Hedbor's avatar
Per Hedbor committed
583

584 585
  - Support Curve25519 key exchange.

586
  - Support client certificates.
Per Hedbor's avatar
Per Hedbor committed
587

588
  - Support ALPN.
Per Hedbor's avatar
Per Hedbor committed
589

590
  - Prefer AEAD suites to CBC suites.
591

592 593 594 595 596 597 598 599 600 601 602 603
  - SSL.File supports set_buffer_mode().

o Standards.PKCS

  Support PKCS#8 private keys.

o String.Buffer & Stdio.Buffer

  Added _search().

o The self testing framework now supports *.test-files.

604 605
o Thread

606 607 608 609 610
  - _sprintf() improvements: Thread.Mutex now prints the ID of the thread
    holding the lock, and thread IDs are shown as hexadecimal numbers.

  - Thread.Farm now supports a callback for thread creation and termination
    for the purpose of tracking thread names.
611

612 613 614 615
  - Thread.Condition are now bound to the first Thread.Mutex object
    that they are used together with. This ensures that the correct
    Thread.Mutex is aways used.

616 617 618 619 620 621
o sprintf %x

  %X and %x can now be used on 8-bit wide strings to get a hexadecimal
  representation of their contents. Just calling sprintf("%x",data) is
  the same as calling String.string2hex(data).

Martin Nilsson's avatar
Martin Nilsson committed
622
o Unicode 10.0.0.
623

624 625
o Unicode.is_whitespace()

626 627
  This new function returns whether a unicode character is a whitespace
  character or not.
628

629

630 631 632 633 634 635
Deprecated symbols and modules
------------------------------

o Sql.mysql_result and Sql.mysqls_result have been deprecated.
  Use Sql.Result instead.

636
o call_function() has been deprecated. Use `()() instead.
637

638

Martin Nilsson's avatar
Martin Nilsson committed
639 640 641 642 643
Removed features and modules
----------------------------

o Compatibility for Pike versions before 7.8 is no longer available.

644
o GTK1 library is deprecated, so glue code has been removed.
Per Hedbor's avatar
Per Hedbor committed
645 646


647 648 649 650 651 652 653 654 655 656
C-level API changes
-------------------

o The contract for functions is now changed so that a function is no
  longer required to clean the stack. The topmost value of the stack
  will be regarded as the return value and the rest of the items on
  the stack, compared to before the function arguments were pushed,
  will be popped and discarded. Efuns still have to clean their stack
  as previously.

657 658 659 660
o Object destructor callbacks may now run during pop_stack() and
  pop_n_elems(). This means that code must be prepared for arbitrary
  state changes after having called them.

Martin Nilsson's avatar
Martin Nilsson committed
661
o Removed add_function, pike_add_function, pike_add_function2,
662 663
  simple_add_variable, map_variable and MAP_VARIABLE. This removes the
  remaining few API:s where text types were used. Use ADD_FUNCTION and
664
  PIKE_MAP_VARIABLE instead.
Martin Nilsson's avatar
Martin Nilsson committed
665

666
o Removed the functions my_rand and my_srand. Use the random functions
667
  on the stack for _random lfuns, the push_random_string or look up
668 669
  the random function from get_builtin_constants(). For deterministic
  pseudo random, create a private Random.Deterministic object.
670

671 672 673 674 675 676 677 678 679 680
o The preprocessor has been converted into a cmod, and been modified
  to use more standard Pike datatypes.

o The preprocessor-specific hashtable implementation has been removed.

o The gdb_backtraces() function is now available also --without-debug.

o There's now support to block mapping hashtables from being shrunk
  on map_delete().

Martin Nilsson's avatar
Martin Nilsson committed
681 682 683 684 685
o guaranteed_memset() is replaced with secure_zero() which fills a
  buffer with zero. On x86 SSE2 is used to zero the memory without
  loading it into the CPU cache, as this function is typically used
  before calling free() on memory with cryptographic key data.

686 687 688 689 690 691 692 693
o The GC marker hash table has been removed. For types which require
  GC markers, they are now allocated as part of the data type. Such
  structures must now start with GC_MARKER_MEMBERS (which includes
  eg the refs field). See "gc_header.h" for details.

  This change significantly improves GC performance (up to a factor
  of 2 in some situations).

694 695 696 697 698 699 700 701 702 703 704 705

Documentation
-------------

o RFC references added.

o Character encoding issues fixed.

o Added @enum/@endenum markup.

o Support undocumented enums with documented constants.

706

Martin Nilsson's avatar
Martin Nilsson committed
707 708 709 710
Building and installing
-----------------------

o GMP 4.1 or later is now required.
Martin Nilsson's avatar
Martin Nilsson committed
711

712 713
o Nettle 3.5 is now supported.

Martin Nilsson's avatar
Martin Nilsson committed
714 715 716 717
o C99 assumed

  The configure tests will not check for functions defined in C99
  anymore and C99 compiler support is assumed.