Rugg/Feldman benchmarks

The Rugg/Feldman benchmarks are a series of seven short BASIC programming language programs that are used to test the performance of BASIC implementations on various microcomputers. They were published by Tom Rugg and Phil Feldman in the June 1977 issue of the US computer magazine, Kilobaud.

The article reported that Integer BASIC, an interpreter program written by Steve Wozniak for the Apple II computer, was much faster than the other programs tested. This sparked widespread comments about the tests, including a lengthy letter from Bill Gates. A follow-up article in the October 1977 issue addressed these concerns in depth and added many new machines and BASICs to the set of results.

John Coll added an eighth test using transcendental functions in an article in the February 1978 issue of the British magazine Personal Computer World (PCW). This expanded set became known as the PCW Benchmarks, and was particularly popular as a test for UK-designed machines like the Grundy NewBrain and BBC Micro.

The benchmark was widely used through the late 1970s, and appears as a standard in many computer magazines and journals. In the 1980s it was not as widely used in the US as the Creative Computing Benchmark or Byte Sieve, but remained in common use in the UK.

History

edit

First tests

edit

The benchmark suite was introduced to test claims that were being made by vendors that their BASIC was much faster than others running on the same machines. The authors were unable to find a set of standardized benchmarks and decided to write their own.[1] The tests deliberately ignored string and floating point performance, as many BASICs of the era, especially those descended from Tiny BASIC, lacked these features. Strings were further hindered by major differences in syntax between different versions.[2][a]

The two ran the benchmarks on every machine they could find, typically at friends' houses. All of the major 8-bit CPUs were tested, including the Intel 8080, Zilog Z80, Motorola 6800 and MOS 6502. For comparison, they also ran it on a CDC Cyber 174 supercomputer, which is so fast that they had to add code to time the run using the system's clock as they could not operate a stopwatch quickly enough. They used this machine for two reasons; one was simply to see how fast such a machine ran, and the other was to ensure no one micro would end up at the top of the list and then claim they had been proven to be the fastest.[3]

The article makes a special note of the BASIC that shipped with IMSAI computers, as it basically didn't work. The machine was very new at the time and the BASIC was described by the company as "preliminary". During this period, different IMSAI machines were delivered with different versions of BASIC. Each version had different features, but none of them were able to run even the majority of the programs.[4]

The Apple II, using the original Integer BASIC, finished well in front of all the other machines. As the tests did not make use of any floating point features, this result was not surprising given the much simpler internal representation of numbers.[b] The next fastest, albeit significantly slower, was Zapple BASIC on a Zilog Z80 add-in card in an Altair 8800. The rest of the list contained a number of very closely spaced entries, dominated by what would later be known as Microsoft BASIC. Across the entire suite of 8080 and Z80 machines and versions of BASICs, the spread was only 20%. In contrast, the 6800-based machines were isolated entirely at the bottom of the list, 30 to 40% slower than the 8080 and Z80 entries.[4]

As the article went to press, the editors of Kilobaud visited several local computer stores to test the programs on newer machines. This process added the production version of IMSAI BASIC, North Star BASIC and the 11 kB BASIC on the Poly-88. IMSAI's results were at the bottom of the pack, while North Star and Poly were in the middle.[5]

Follow-up

edit

Rugg and Feldman revisited the suite in the October 1977 issue. The new article opened by noting they had been deluged with mail about the original article. Among them was a letter from Bill Gates, who they introduce as the author of "Altair BASIC (8080 and 6800 versions), OSI BASIC and PET BASIC". The last entry refers to a member of the "1977 trinity" of machines, the Commodore PET, which was made available in prototype form for inclusion in the article.[6]

Gates complained that the original test series "let an integer BASIC be compared against... more powerful BASIC... using floating point." He suggested that the test include a DEFINT A-Z at the start, which would make newer versions of Altair BASIC use integer math as well. Instead, Rugg and Feldman took another approach and eliminated all integer-only BASICs from the new test results.[6] This did not eliminate the Apple II, which by this time had introduced the MS-derived Applesoft BASIC on cassette.[7]

Gates also noted that the results for the 6800 machine were not indicative of this processor. The machine they used, the Altair 680, runs the CPU at half its rated speed. He suggested a more typical 6800-based machine would be slightly faster than the 8080. However, the 6800 once again put in a poor showing even on newer machines running at higher speeds.[6]

Gates also suggested that the Cyber 173's time was likely due to it being a compiler rather than an interpreter. The authors point out this was not the case for the TRW BASIC they used in the original test, and then use this as a segue to compare the differences between compilers and interpreters.[6]

Gates concluded his letter by noting that a number of the results were identical in spite of being on different machines. He suggests this is because the BASICs in question contained "signatures" from Altair BASIC and are thus "illegitimate software". He did not specify which ones he claimed were stolen, and the authors responded by saying that if "Bill can stop people from selling them through legal means, we'll stop listing them."[8]

Rugg and Feldman conclude the article by mentioning some of the other concerns that were raised after the original article. One common issue was the lack of more advanced mathematical functions, which they acknowledge, but suggest this is something best left to the reader. The other was the lack of string manipulation, but they note that the syntax of string handling differed considerably between platforms and thus could not be made in a single version.[8][c]

In this series of tests, the list was topped by the OSI Challenger, a 6502-based machine that had been "souped up" to 2 MHz, double that of typical 1 MHz 6502 machines of the era like the Apple II and PET. When running at its normal 1 MHz speed, the Challenger was just beaten by Zapple BASIC on Z80 machines running at 4 MHz. PET BASIC was next, only slightly behind the Challenger.[9] They conclude that the 6502 is the highest performing of the CPUs, agreeing with comments Gates had made in his letter. The 6800 once again ends up in last place.[10]

PCW version

edit

As part of a longer article discussing new entries into the computer market, including the TRS-80, John Coll used the Rugg/Feldman tests to benchmark a variety of machines available to him in the UK in October 1977. He added an eighth test to exercise the math routines, and provided the resulting run times both on their own as well as the additional time compared to Test 7, in keeping with the earlier concept of each test modifying the last.[11] The results were published in the first issue of Personal Computer World in February 1978,[12] with a short follow-up in their November 1978 issue.[13]

As one of the earliest sets of BASIC benchmarks, the F/R tests were seen primarily in the late 1970s and early 1980s. It was a standard among reviews in Kilobaud, used to compare the many new varieties of BASIC that continued to appear for early microcomputers.[14] Compute! used it for their 1979 review of the Challenger 1P,[15] and 68 Journal used it to demonstrate the extremely high performance of BASIC09.[16] InfoWorld used it for their 1981 review of a new BASIC for the TRS-80,[17] and the TRS-80 Color Computer as a whole.[18]

After that point the Byte Sieve began to become popular and the number of articles referring to the F-R benchmark become less common, but it could be found even such rarefied sources as the HP Journal.[19]

The PCW versions remained very popular in the UK, and can be found in many reviews of UK-centric machines like the Grundy NewBrain, various Sinclair Research machines,[20] and the BBC Micro.[21]

Programs

edit

The programs were designed to allow a user to type in the first test, run it, and then modify it in-place to run the subsequent tests. This meant the user did not have to type in seven different programs, but simply modify a single one.[3] The first seven listings are the originals from the 1977 article,[1] the eighth is the PCW addition.[22]

Benchmark 1

edit
300 PRINT"S"
400 FOR K=1 TO 1000
500 NEXT K
700 PRINT"E"
800 END

FOR-NEXT loops are one of the most fundamental constructs in the BASIC language, and if the performance of these loops is slow it is highly likely that any program running in that BASIC will be slow as well. A famous example of this is Atari BASIC, which had several problems that greatly slowed the performance of FOR-NEXT loops compared to contemporary examples, and BASIC programs on the Atari were generally very slow as a result.[23]

Benchmark 2

edit
300 PRINT"S"
400 K=0
500 K=K 1
600 IF K<1000 THEN 500
700 PRINT"E"
800 END

This test ultimately performs the same operations as test 1, but in this case, it uses an explicit test and jump rather than using the built-in FOR-NEXT construct. Generally, this program runs much slower than 1 because most BASICs parse the parameters in the FOR, including the line number, only once when it is first encountered. Using the IF, as in test 1, causes it to parse the values every time through the loop. Moreover, most BASICs do not simply store the parsed line number of the top of the loop, but the memory address, whereas a THEN requires the interpreter to scan through the program for the corresponding line number, in this case 500. Although this test does not show it due to its small size, this search takes increasingly long times as the program length grows. Some versions of BASIC optimized GOTO using explicit labels,[24] or pushing the GOTO targets on a stack to make them perform like NEXT.[25] Turbo-Basic XL did both, and ran much faster than any similar BASIC as a result.[24]

Benchmark 3

edit
300 PRINT"S"
400 K=0
500 K=K 1
510 LET A=K/K*K K-K
600 IF K<1000 THEN 500
700 PRINT"E"
800 END

Test 3 is an expansion of test 2, this time adding some basic mathematics and variable access. By comparing the time to run tests 2 and 3, one can get an idea of the performance of the language's math library.[1]

Benchmark 4

edit
300 PRINT"S"
400 K=0
500 K=K 1
510 LET A=K/2*3 4-5
600 IF K<1000 THEN 500
700 PRINT"E"
800 END

The same as test 3 except that the variable K is replaced by numeric constants. This requires the interpreter to convert the values from their textual representation into their internal storage format, which takes time (unless this is performed once and for all, before execution, as in some more advanced BASICs). Some idea of the performance of this conversion functionality can be determined by comparing this time to benchmark 3.[1]

Benchmark 5

edit
300 PRINT"S"
400 K=0
500 K=K 1
510 LET A=K/2*3 4-5
520 GOSUB 820
600 IF K<1000 THEN 500
700 PRINT"E"
800 END
820 RETURN

Test 5 introduces a subroutine call. Long programs in early versions of BASIC would make extensive use of subroutines and thus the efficiency of the calling mechanism was important. Depending on the way the system worked, the return might cache the location of the calling line in a fashion similar to NEXT, and thus run very quickly. Others might store the line number of the calling line, and thus require the code to scan the program listings to RETURN.[1]

Benchmark 6

edit
300 PRINT"S"
400 K=0
430 DIM M(5)
500 K=K 1
510 LET A=K/2*3 4-5
520 GOSUB 820
530 FOR L=1 TO 5
540 NEXT L
600 IF K<1000 THEN 500
700 PRINT"E"
800 END
820 RETURN

Test 6 defines a small array at the start and adds another FOR-NEXT loop inside the main loop. This has little effect on the code, but is used to set a baseline for test 7.[1]

Benchmark 7

edit
300 PRINT"S"
400 K=0
430 DIM M(5)
500 K=K 1
510 LET A=K/2*3 4-5
520 GOSUB 820
530 FOR L=1 TO 5
535 M(L)=A
540 NEXT L
600 IF K<1000 THEN 500
700 PRINT"E"
800 END
820 RETURN

This assigns a value to each of the array elements every time through the loop. Comparing the time needed to run 7 to 6 indicates the efficiency of array access.[1]

Benchmark 8

edit
300 PRINT"S"
400 K=0
500 K=K 1
530 A=K^2
540 B=LOG(K)
550 C=SIN(K)
600 IF K<100 THEN 500
700 PRINT"E"
800 END

Test 8 was added by PCW,[11] performing several transcendental functions in order to test their performance. This code does not include the code from test 7, which breaks the original pattern of each test adding to the last. Instead, the associated article listed both the time to run 8 and 7 and 8 combined.[12]

Sample results

edit

This list is not meant to be exhaustive, but instead a quick overview of the results seen on some popular systems of the early microcomputer and home computer era. The first table includes a selection of machines from the original tests in June 1977, with exceptions as noted.[1] Test time is in seconds.

System CPU BASIC Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 Test 8 Notes
Apple II 6502 @ 1 MHz Integer BASIC 1.3 3.1 7.2 7.2 8.8 18.5 28.0 PCW's numbers are about 0.1s longer
Apple II 6502 @ 1 MHz Applesoft BASIC 1.3 8.5 16.0 17.8 19.1 28.6 44.8 55.5 from PCW, for comparison with above
Altair 8800 8080 @ 2 MHz Altair BASIC 4.0 1.7 10.2 21.0 22.5 24.3 36.7 52.4
Altair 8800 8080 @ 2 MHz Altair BASIC 4.0 1.9 7.5 20.6 20.9 22.1 38.8 57.0 67.8 from PCW, including FP
Altair 8800/TDL ZPU Z80 @ 4 MHz Zapple 1.7 9.5 20.6 21.7 23.7 36.7 52.4 Z80 add-on board, Zapple BASIC shipped with it
Compucolor 8001 8080 @ 2 MHz Compucolor 8k BASIC 2.1 13.1 27.0 29.0 31.3 47.5 67.8
SWTPC 6800 6800 @ 898 kHz SWT 8k BASIC 1.4 9.0 16.8 18.1 20.0 31.0 45.2 53.7 from PCW
SWTPC 6800 6800 @ 898 kHz Altair 680 BASIC 14.9 24.7 96.1 105.3 109.8 174.1 204.5
Altair 680 6800 @ 500 kHz Altair 680 BASIC 1.1 2.6 16.4 30.9 33.7 36.6 56.0 81.9 96.9 from PCW
System CPU BASIC Test 1 Test 2 Test 3 Test 4 Test 5 Test 6 Test 7 Test 8 Notes
IBM 5100 4.5 21.1 57.4 54.5 59.0 88.2 174.9 201.6
BBC Micro 6502 @ 2 MHz BBC BASIC 0.8 3.1 8.1 8.7 9.0 13.9 21.1 49.9 fastest of the English/American 8-bit BASICs
Acorn Electron 6502A @ 2 MHz BBC BASIC 1.11 4.01 11.12 11.76 12.40 18.72 28.71 72.53 from PCW [26]
Atari 800XL 6502 @ 1.79 MHz Atari BASIC 2.2 7.3 19.7 24.1 26.3 40.3 60.1 na no explanation for lack of test 8
Commodore 64 6510 @ 1 MHz MS BASIC 1.2 9.3 17.6 19.5 21.0 29.5 47.5 119.3
ZX80 Z80 @ 3.25 MHz Sinclair BASIC 1.5 4.7 9.2 9.0 12.7 25.9 39.2 NA integer only
ZX Spectrum Z80 @ 3.58 MHz Sinclair BASIC 4.4 8.2 20.0 19.2 23.1 53.4 77.6 239.1
Acorn Atom 6502 @ 1 MHz Atom BASIC 0.5 5.1 9.5 10.8 13.9 19.1 31.1 92 from DatorHobby 1/84[27]
Luxor ABC 80 Z80 @ 3 MHz ABC BASIC 1.1 2.3 11.1 12.1 12.6 17.7 23.9 136 from DatorHobby 1/84
ABC 80 (using integer variables instead of floating point) Z80 @ 3 MHz ABC BASIC 0.3 1.1 3.5 3.5 3.6 5.8 9.3 65 from Mikrodatorn 1982
ABC 800 (using single precision floating point) Z80 @ 3 MHz ABC BASIC II 0.9 1.8 6.0 5.9 6.3 11.6 19.6 29 from Mikrodatorn 1982
ABC 800 (using double precision floating point) Z80 @ 3 MHz ABC BASIC II 1.2 2.2 10.0 10.6 11.0 17.8 26.4 144 from Mikrodatorn 1982
VIC 20 6502 @ 1 MHz MS BASIC 1.4 8.3 15.5 17.1 18.3 27.2 42.7 99 from DatorHobby 1/84
Dragon 32/64 6809E @ 0.89 MHz MS Extended BASIC 1.6 10.2 19.7 21.6 23.3 34.3 50.0 129 from DatorHobby 1/84
Spectravideo SV-328 Z80 @ 3.6 MHz MS Extended BASIC 1.6 5.4 17.9 19.6 20.6 30.7 42.2 236 from DatorHobby 1/84
TI-99/4A 9900 @ 3 MHz TI BASIC 2.9 8.8 22.8 24.5 26.1 61.6 84.4 382 from DatorHobby 1/84
Oric Telestrat 6502 @ 1 MHz Oric BASIC 0.5 3.6 11.4 13.2 13.4 18.7 26.3 109 Cartridge BASIC, measured by community member
Oric Atmos 6502 @ 1 MHz Oric BASIC 1.1 1.6 15.2 25.4 27.4 33.0 45.6 68.5 136 measured by community member
Oric 1 6502 @ 1 MHz Oric BASIC 1.0 1.8 17.1 29.0 31.4 38.0 51.8 77.8 230 measured by community member
Tangerine Microtan 65 6502 @ 0.75 MHz MS Extended BASIC 1.9 12.8 24.7 27.8 29.6 43.2 68.9 243 with TANEX expansion? Measured by community member.
VTech CreatiVision 6502 @ 2 MHz CreatiVision BASIC (1983) 20 60 79 119 180 298 398 120 executes in VRAM[28]
VTech Laser 2001 6502 @ 2 MHz VTech Laser 2001 BASIC 1.7 6.7 11.6 12.9 15.3 26.0 39.7 64.5 same hardware as CreatiVision but executes in CPU RAM

Notes

edit
  1. ^ At the time there were two major dialects of string handling used in BASIC. The first had been introduced by Hewlett-Packard as part of their HP 2000 line and used array slicing for manipulation. The second had been introduced by Digital Equipment (DEC) and used string functions instead. Microsoft BASIC, based on the DEC dialect, became the de facto standard and by the late 1970s the earlier syntax had largely disappeared with the exception of Atari BASIC.
  2. ^ Integer BASIC used 16-bit binary numbers, while most BASICs used 32- or 40-bit floating point formats, which requires longer to process. See Microsoft Binary Format for details.
  3. ^ The issue with string syntax stems from the difference between BASICs that were inspired by DEC's vs. HP's dialects. Microsoft BASIC was based on DEC's BASIC-PLUS variety and uses the MID/LEFT/RIGHT functions to manipulate strings, while HP used an "array-of-chars" system more similar to FORTRAN 77 and C that is found in Atari BASIC, North Star BASIC, and others. The two are different enough that porting directly from one to the other is not trivial, and such portability was a goal of the benchmark suite.

References

edit

Citations

edit
  1. ^ a b c d e f g h RuggFeldmanJun1977, p. 66.
  2. ^ RuggFeldmanJun1977, p. 67.
  3. ^ a b RuggFeldmanJun1977, p. 68.
  4. ^ a b RuggFeldmanJun1977, p. 69.
  5. ^ RuggFeldmanJun1977, p. 70.
  6. ^ a b c d RuggFeldmanOct1977, p. 22.
  7. ^ Herzfeld, Andy (12 February 2014). "MacBasic – The Sad Story of MacBasic". Folklore.org.
  8. ^ a b RuggFeldmanOct1977, p. 24.
  9. ^ RuggFeldmanOct1977, p. 23.
  10. ^ RuggFeldmanOct1977, p. 25.
  11. ^ a b PCWFeb1978, p. 57.
  12. ^ a b PCWFeb1978, p. 58.
  13. ^ PCWNov1978.
  14. ^ Didday, Rich (January 1978). "A tale of Four BASICs; which one for your 6800?" (PDF). Kilobaud. pp. 50–59.
  15. ^ Russell, W. Keith (Fall 1979). "The Challenger 1P". Compute!. pp. 100–104.
  16. ^ Harmon, Tom (September 1980). "OS/9-BASIC09". 68 Micro Journal. pp. 26–27.
  17. ^ Puckett, Dale (31 August 1981). "Extended Color BASIC for TRS-80 from Microsoft". InfoWorld. pp. 26–28.
  18. ^ Puckett, Dale (16 August 1982). "Radio Shack Color Computer". InfoWorld. pp. 57–61.
  19. ^ Bidwell, John; Palermo, David (May 1982). "Advanced Multilingual Computer Systems for Measurement Automation and Computer-Aided Engineering Applications" (PDF). Hewlett-Packard Journal. 33 (5): 3–7.
  20. ^ Personal Computer World, 1983 and 1984 Benchtest Summaries
  21. ^ Ochocki, Peter. "Personal Computer World Benchmark Tests".
  22. ^ PCWNov1978, p. 58.
  23. ^ Wilkinson, Bill (1982). Inside Atari DOS. Optimized Systems Software. p. 191. ISBN 0-942386-02-7. Retrieved 2009-04-04.
  24. ^ a b "Turbo-Basic XL" (PDF). Happy-Computer. December 1985. pp. 34–47.
  25. ^ BASIC XL (PDF). Optimized Systems Software. 1983. p. 35.
  26. ^ Benchtest:Acorn Electron. October 1983.
  27. ^ Dator Hobby Nr1 1984.
  28. ^ CreativEmu Forum thread.

Bibliography

edit

Further reading

edit