Gyassa Software Blog

The numbers 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 12, 13, 15, 16, 18, 21, 22, 24, 25, 28, 30, 33, 37, 40, 42, 45, 48, 57, 58, 60, 70, 72, 78, 85, 88, 93, 102, 105, 112, 120, 130, 133, 165, 168, 177, 190, 210, 232, 240, 253, 273, 280, 312, 330, 345, 357, 385, 408, 462, 520, 760, 840, 1320, 1365, and 1848 are called Idoneal numbers. If you do a google search on “Idoneal Numbers” you will find many useful links, so I will not put any links defining Idoneal numbers here. The Idoneal numbers are also called convenient numbers for the reason we will be explaining below. Also, it is mostly assumed that this is the complete list, but there is a possibility that there is one more undiscovered Idoneal number.

Our focus is on the relationship between Idoneal numbers and how primes behave when you try to split them into a square plus a square times an idoneal number. In particular we are looking at the expression 

 x² + d·y²

for varying values of d.

This leads us to the following question. For a given d, is there a simple characterization of a prime p on whether p can be expressed in such an equation with x and y being positive integers? For example, when d is 5, a prime can only be expressed by such an equation if p is congruent to 1 or 9 mod 20. So to be more specific, for a specific value of d, can we determine the primes expressible by the equation by whether they satisfy a congruence relation?

The answer, which is probably not surprising at this point, is that the list of Idoneal numbers is exactly the numbers d where such a congruence relation is possible. That makes these numbers quite "convenient" when trying to predict when p can be expressed by the equation above. But there is more that we can say.

Before asking whether the equation has an exact expression for a prime p, you can ask whether you can express p if you do arithmetic mod p. This is equivalent to asking if -d is a square mod p, or what is called a quadratic residue. Or using the language of the Jacobi symbol that is used to calculate quadratic residues, it is asking if the Jacobi symbol of -d over p has the value 1. Using quadratic reciprocity it is possible to characterize whether -d is a square mod p by congruence relations of p mod 4·d or sometimes mod a number that divides 4·d. All this can be efficiently calculated by a computer, and for Idoneal numbers with pencil and paper if you are good with arithmetic. We will not go into the details here, but the start of practically any book on number theory will go through how to do such a calculation.

So we can easily characterize p for which the expression has solutions mod p. We need to be able to lift it to that characterization to the integers. Is there a general way to do this? 

The answer is that d has to be an Idoneal number, as said before, and for each odd prime q that divides d, p must be a square mod q. See Primes of form x^2 + ny^2 and congruences and the statement towards the end.  

This can be calculated using quadratic reciprocity. Let us look at 15 as an example. First we need -15 to be a square mod p. Using quadratic reciprocity we can show that it breaks down into two cases. The first is for p = 1 mod 4, for those p, p must be congruent to 1, 2, 4 or 8 mod 15. The second is for p = 3 mod 4 where it must be 7, 11, 13 or 14 mod 15. Also, since 3 and 5 are the two odd primes dividing 15, p must be a square mod 3 and a square mod 5. This essentially reduces us to the p = 1 mod 4 case with 2 and 8 being eliminated. Or expressing this mod 60, p must be either 1 or 4 mod 60. For the calculation done for all Idoneal numbers see Congruences for primes that can be expressed as square plus an Idoneal number times a square. This list was not actually computed using quadratic residues but by brute force using variations of the script referenced in the Readme for my recent work with primes.

There is also another characterization of an Idoneal number. If d is an Idoneal number then every square of a prime p, assuming that p does not divide and -d is a square mod p, satisfies the equation above non-trivially, i.e. with y != 0. The converse is also true. If d is not an Idoneal number, then there are many squares of primes that cannot be expressed this way even if -d is a square mod p.

Since Idoneal numbers have a simple congruence for determining whether p can satisfy the squares expression, it is possible computationally to find primes for many Idoneal numbers simultaneously. In fact, it is possible to find a prime that can be represented as a square plus an Idoneal number times a square for all the Idoneal numbers simultaneously. See First prime expressed as x^2 + d·y^2 for all Idoneal d.

I am not the only one to think 73 is special. See 73_(number) for a Wiki page on 73. But I have my own particular reason for seeing it as special.

We start with a question. But before get there, we start with some definitions. For integers N larger than 1, consider the integers mod N. Or to put it another way, map integers to their remainders after dividing by N. The remainder will be a number between 0 and N - 1. You can add and multiply these remainders and take the remainder of the result. This creates well defined addition and multiplication mod N. Now, of these remainders, only take the non zero remainders that have no common factor with N. For example, if N is 6, then only 1 and 5 satisfy this condition. In mathematics, these remainders under multiplication are called the multiplicative group mod N. It is group because it has an operation, multiplication, which combines two elements of the group to produce another element. Now in this multiplicative group, some elements have the property that if you square it, i.e. multiply it against itself, it gives the remainder 1, or the identity element of the multiplicative group. As a reminder, the identity element is an element that when multiplied against any other element leaves that element untouched.  We will call the remainders whose square is the identity to be of "order 2". Now notice that for N = 8, all the non identity elements in the multiplicative group, which are the remainders 3, 5, and 7, are of order 2. This raises the question, what other N is this true for? Or to be more precise, the question is this, what is the largest N such that the every remainder mod N that has no common factor with N and is not equal to 1 has order 2?

Why is such a question of interest? In mathematics group elements of order 2 are associated with symmetries. For example, if you flip a card and then flip it again, you have acted on it by group action of order 2. Symmetries in mathematics and physics show up everywhere and tend to be fundamental in the theories. Specifically for physics, symmetries, which are also dualities, are foundational. At every place you look you find examples, such as positive and negative charges, up and down spin states for particles, clockwise vs. counterclockwise rotation, and on and on. The derivation of special relativity uses symmetries as its primary driving assumption. They are also at the foundation of a lot of the richer theories in mathematics as well. In our particular case, suppose we have determined the N that is the answer to the question, would primes that map to the identity element mod N have any special properties? And what does this have to do with the prime 73? If you have been reading any of the prior blog articles, you may be able to guess the answer. An answer can be found at Largest N with All Order 2 Multiplicative Group Mod N.

This is a follow up of Three Cases of Primes From Squares which is itself a follow up of prior blogs articles. I started this journey by recalling the theorem that all primes congruent to 1 mod 4 can be written as a sum of two squares. I then recreated the algorithm for efficiently computing the two squares using the Euclidean algorithm on the Gaussian integers. This then led me to examine finding primes and their decompositions into integer squares for other d in the expression p = d·a² + b² where p is a prime. The cases d = 1 and d = 4 are equivalent, so I started with the cases of d = 2 and d = 3 where the quadratic imaginary extension produced by adding the square root of -d was still Euclidean. At that point I thought that I was done. I did not, at first, see a practical way to find the a and b for arbitrary values of d. But, as what is typical for me, I did not know how to stop picking at the problem. One day, when I figured out that there was actually an efficient algorithm for finding square roots mod p, I saw a way forward. But the main breakthrough was when I discovered that the Euclidean algorithm could be extended to non Euclidean fields in a useful way. Investigating this approach, I ran into another roadblock problem that I describe in Approximating Integers with Multiples of Rational Numbers. After eventually finding an efficient solution to that problem, I had all the pieces for breaking down primes with 100s of digits into d times a square plus a square for d up to about 10,000 in less than a minute on my laptop.

However, once I had code that could do arbitrary d, I wondered about finding primes that could do many d simultaneously. For example, what is the first prime that can be written as two times a square plus a square, three times a square plus a square, all the way up to 10 times a square plus square? The answer, which you can get if you execute the code in the link to the Groovy Console below, is 1009. Note that this prime is not as large as you might think it should be. In researching this general problem, I found some other interesting behaviors, which I will describe in blog articles following this one.

The Groovy code for the arbitrary and many simultaneous d can be found at the Gist Arbitrarily split primes into a multiple of a square plus a square. A preloaded and executable version can be found at Groovy Web Console. When executed it may take time for the code to be compiled. Results for simpler cases usually return in under 10 seconds. And as described in earlier blog articles, the Result tab is where you can find the primes and their decompositions. If you are truly interested in reading and understanding the code, you should probably start with prior iterations in the earlier blog articles. 

When doing larger d, one issue immediately comes up. The solutions you produce are for multiples of the prime, not just for the prime itself. As an example, if you examine the case of d = 5, the prime 7 cannot be written a 5 times a square plus a square. Instead it is 14 = 2·7 that can be written as one squared times 5 plus three squared. I deal with that by filtering out primes that have such solutions and only keeping the ones where p = d·a² + b² with no multiple of p. But the code has a flag where this filtering can be turned off.

This article is motivated by the work done for prior blog articles on splitting primes into sums of squares. Lately, I have been wanting to extend that work to compute the splitting of primes into d·a² + b² for arbitrary positive integers d instead of limiting myself to d equal to 2, 3, and 4. As part of that work, I found I needed to solve the following problem.

Given a positive integer b and an arbitrary rational number find an integer in the range from 1 to b - 1 (from 1 to one less than b) such that multiplying by that number will yield a number that is within 1/b of an integer, either below or above. In the practical scenario that I envision, b tends not to be that large while the rational number can have a numerator and denominator with hundreds of digits.
In the following assume % is the operator that computes the remainder after division and · represents taking the product. Then the problem posed above is equivalent to the following problem. 

 Given arbitrary positive integers p, q, and b with b > 2,
 find a number m in the range {1..b-1} such that
 if we set r = (p·m·b) % (b·q), then r ≤ q or r ≥ (b - 1)·q.

In this statement, the rational number being approximated to an integer is p over q.

The fact that such an m exists has an interesting proof, which I will not do here, I will leave it as an exercise for the reader. Our focus is on efficiently finding the m.

There are two ways to view this problem. Given a particular b, then with preparation work allowed for that b, implement an efficient algorithm for any p and q. The other approach is given any p, q, and b find a general efficient algorithm. I have had some success with the first view of the problem. For the second, I am not able to find an algorithm that is significantly faster than trying every possible multiple from 1 to b - 1.

For the first view, I have implemented a Groovy script that can be found at https://groovyconsole.dev//?gist=sampwhite/94a1657599bf049073b42156fc18b75d#MDujiDMMSL. It works well with b up to 1,000 and can be stretched to work with b up to 10,000 if you are willing to wait a few minutes. I will be incorporating this work into my next version of code that splits a prime into b times a square plus a square.

For the second view, I have been researching the problem with Google with little success. It feels like there should be some type of Euclidean algorithm style solution that will allow me to do useful computations with b greater than 10,000, but I have not found anything. The closest analogous problem I have found is on Diophantine Approximation, but that approach does not allow for multiplying by an integer as part of their construction.

Part of my struggle, which I have referenced before in prior blog articles, is coming up with the correct words to characterize the problem.  Is it a real number approximating an integer problem, a problem in calculating the class group in quadratic imaginary number fields, or a prime ideal calculation problem in number fields? When I have researched these topics, either there is no interest in multiplying the input number by a small integer before approximating an integer, or the mathematics is very advanced and uninterested in algorithmic implementations.

In the two prior blogs I have explored decomposing primes into two squares. In the first prior blog, I focused on just breaking down primes congruent to one mod 4 into the sum of two squares. In that article I produced a Groovy script to do that breakdown. In the second prior article I extended it to primes congruent to one mod 3 and the decomposition into 3 times a square plus a square. In that blog article I also said this was the other efficiently computable case. That implicitly claimed that I had exhausted the cases that had fairly straightforward efficient algorithms for computation. That got me to thinking, is that really true? Are there any other cases that are efficiently computable with fairly simple algorithms?

In the Wiki Article on Fermat Squares, it states that if a prime is congruent to 1 or 3 mod 8, it can be broken down into two times a square plus a square. Looking at that, I wondered if I could handle that case as well. It involves the number field obtained by adding the square root of negative two, and that is also an Euclidean Domain. So the only real issue was finding a multiplicative square root of -2 in the integers mod p. And I remembered that if i is the square root of negative one, then (i + 1) * (i + 1) = 2i. From that initial thought, I was able to piece together an algorithm for this case.

In what follows I give a synopsis of the math that I used for this case. One of the rules I used is that I would not be allowed to consult any text book or article for the actual math, even though there is quite a nice textbook that you can find at Primes of The Form x² + ny², and instead I would rely on the math I learned over 30 years ago.

The case of primes congruent to 1 mod 8 is fairly straightforward since eight divides the order of the multiplicative group mod p. That means in multiplicative group mod p, there is an eighth root of unit and if you have a generator for the multiplicative group, then taking that generator to (p - 1)/8 power will give you that eighth root of unity. Let w be that root, then some simple math shows that the square of w(1 + w²) is -2. Note that w² is the i from the earlier paragraph.

The case of primes congruent to 3 mod 8 is a bit trickier and involves extending finite fields. In this case, the multiplicative group of non-zero numbers mod p, split into two equal groups, those that are a square of a number mod p, and those that are not. And those that are a square form a subgroup of the multiplicative group mod p. Also, there is no fourth root of unity, which means that -1 is not a square mod p. So you can get from the subset of squares to non squares (and vice versa), just by multiplying by -1. If you extend the field mod p by the square root of -1, you will get a field of order p². The multiplicative group for that is of order p² - 1, and given that p is congruent to 3 mod 8, it is not hard to see that the order of this group is exactly divisible by 8 (if you understand the phrase “field mod p”, you should then be able to do this calculation as well). That means that in this new field, it has an eighth root of unity but not a sixteenth root of unity. Now, with a bit of mathematics of how you construct a sixteenth root of unity from an eighth root of unity, it is not difficult to show that this means that 2 cannot be a multiplicative square, which means that -2 is a square. However, a number not being a square mod p happens if and only if that number to the (p - 1)/2 power is -1. So 2 to the power (p-1)/2 is -1. Then since (p+1) is divisible by 4, some simple math shows that 2 to the power (p+1)/4 is a square root of negative 2.

In the prior two blog posts, I put the Groovy script into the body of the blog post. The Groovy has now gotten sufficiently complex that this time I decided to create a Gist and reference that instead. The Gist is at ​​https://gist.github.com/sampwhite/e58bbb44595bfb7d056de0578a2b9e3e. And the Groovy Web Console pre-loaded with the Gist can be found at https://gwc-experiment.appspot.com/?gist=sampwhite/e58bbb44595bfb7d056de0578a2b9e3e. Also, unlike in the previous iterations, the code is written as a more formal code construct. Class and method names are longer, there is a bit more commentary, and the code is no longer treated simply as a script. And in one other change, for the case of primes congruent to one mod four, I wrote the breakdown into squares as 4 times the square of a number plus the square of the number. You can do this, because if an odd number is the sum of two squares one of the squares is a square of an even number. By doing this we can make a stronger claim of uniqueness. Each of the breakdown into squares is the unique breakdown into nx²+y² for a given n and possible positive integers x and y. As before, in the Groovy Web Console Result tab, the case of two times a square plus a square is prefixed with M2, the case for three times a square plus a square is prefixed with M3, and the case of four times a square plus a square is prefixed with M4.

And for one last thing (at least for this blog article), I ran the code as an application on my laptop and in less than two minutes generated a prime with 1,000 digits congruent to 1 mod 24 that exhibited all three breakdowns into sums of squares. The example can be found at https://gist.github.com/sampwhite/c84cfa1fe0718d54d5f75ce3219ddd0c. I find this to be remarkable. A number with a hundred digits is already larger than all the atoms in the universe. Here is a prime number incredibly larger in order of magnitude that we claim is uniquely broken down into particular combinations of squares of integers, and we are able to compute that breakdown.

This is a follow up of the prior blog post on primes that are congruent to 1 mod 4 being uniquely the sum of squares. In other words, such primes can be written as x² + y² with x and y positive integer. There is a natural follow up question, are there any more ways to combine squares of integers to produce primes. And yes, there is one more easily computable example. If a prime is congruent to 1 mod 3, then it can be written uniquely as the three times the square of one integer plus the square of another. In other words, such a prime can be written as 3x² + y² with x and y positive integers. And like for primes congruent to 1 mod 4, there is an efficient way to compute the break down of the prime into the squares. For more on this general topic see Fermat Sum of Two Squares.

For the primes congruent to 1 mod 4, we used the Gaussian integers and applied the Euclidean algorithm. For the primes congruent to 1 mod 3, we use the Cyclotomic 3rd root of unity integers and still use the Euclidean algorithm. Note, the Gaussian integers are actually the Cyclotomic 4th root of unity integers. See Cyclotomic Fields Wiki for a definition of Cyclotomic fields and the definition of a ring of integers inside the field. 

The key fact is that Cyclotomic 3 and Cyclotomic 4 fields are quadratic extensions of the rational numbers. And the general theory of Quadratic Forms says that finding solutions mod a prime will, for enough primes and powers of primes, will give you a solution in the integers. For our two simple cases, finding a solution mod p is sufficient for a mod p case to be lifted to integer solutions.

The code for the case of primes congruent to 1 mod 3 is similar enough to the code for primes congruent to 1 mod 4, that instead of providing code just to do the 1 mod 3 case, the code does both cases simultaneously. In particular if a prime is congruent to 1 mod 12, you will see a break down into a sum of squares and the breakdown into three times a square plus a square. To see the code click the three dots at the end of this sentence. ...

Lately I been revisiting some of my old number theory that I learned decades ago. In particular, there have generally been two principles that I learned about numbers.

• Primes are essentially random and at numbers of around size N, they appear with density one over the natural logarithm of N.
• Squares of positive integers are not random. And the density of squares at size N is one over the square root of N, far less often than primes for large N.

So decades ago, I was amazed by the theorem that says every prime congruent to one mod 4 (or have remainder one when dividing by 4) is always the unique sum of two squares. To see more click the triple dots following this sentence. ...

For those who have read prior articles in this blog, you will see an emphasis on using data to drive application design. And when I say data, I do not mean data stored in database tables, but data created using code or stored in files in source code. Recently, I started trying to look for other developers that may have come to similar conclusions to my own, but I have yet to find a kindred soul. As I explained in the prior blog, this is due to the ambiguity in terms like schema, runtime, and dynamic making it impossible to create a good google search to find people with similar ideas to my own. 

I got so frustrated, that I asked the question, has anybody registered a domain name using a combination of my search terms? For example, picking two of the most generic of my terms, surely somebody has created a website with a domain name that joins the words dynamic and runtime. So I looked to see if there was a website with domain name dynamicruntime.com or dynamicruntime.org and I was shocked to discover that nobody had yet thought to register this domain name. So, in a challenge to the world at large, I have registered both dynamicruntime.org and dynamicruntime.com under the ownership of Gyassa Software and I have set up a website at https://dynamicruntime.org.

Fortunately, during the last few months, I have had the opportunity to focus exclusively on my own projects. During these months I decided to investigate AWS and all the amazing things it allows software developers do for free. As part of that effort, I created the github account https://github.com/sampwhite/dynamicruntime where I have put my recent labors. It is all under the MIT license, so feel free to steal or copy any of it. This repository is the code base for the website that is now deployed at dynamicrruntime.org.

The thing I like most about this project is the speed with which the server starts up. By eschewing the standard large frameworks (such as Spring) and writing endpoint implementations from scratch, I am able to keep the start up time under five seconds. And this is for an application that automatically creates database tables and implements a full login implementation that supports two step authentication using email. The code base has other nice characteristics that come from avoiding the usage of third party frameworks. For example the method call stacks are fairly shallow and IntelliJ just loves the code.

If you have read the prior articles in this blog on runtime data differencing, one of the obvious themes is that code should be turned into data whenever it is reasonable to do so. When you take this approach to doing coding and apply it to schema, you find yourself at odds with popular practice. Popular practice is to define schema at compile time. You use reflection to dig through the definition of classes and pull in all the annotations. This is static schema where the entire design of endpoints or tables are defined directly as hardwired artifacts in the code itself. Currently the most dominant version of this approach for Java is Spring Boot and Hibernate. The approach is popular because it has a quick on-ramp to getting a working solution and it, at least initially, can eliminate a lot of boilerplate code. I should say that there are many projects where using compile time schema makes a lot of sense. However, I wish to focus on projects where such an approach does not make sense, since from my google searching, there is little discussion on this point. Click on the triple dots below to see more.
...

Overview

Over my many years (actually decades now) of my time as a programmer, one of the favorite discussion among programmers is on the relative merits of various programming languages. In this particular post, I thought I would write a persistent record of my thoughts on the matter.

Before we start, let me begin with a list of languages in which I have written a piece of code of some significance. I will give them in the order that I encountered them. I will exclude from this list languages that have a very particular use, such as SQL or Bash, even if such languages have grown up to have capabilities outside their original mandate. The languages are Basic, Fortran, Assembly, C,  Pascal, C++, Perl, Java, Javascript, Smalltalk, Groovy, Scala, and Python. I have encountered Lisp in a text book setting and have only written Lisp in such a setting. Also, in my usage of Javascript I did quite a bit of work in HTML and CSS, the natural domain for Javascript.

Static vs Dynamic

To start, I classify languages into two camps, dynamic and static. In a dynamic language, the compiler does little to help you if you write a method call with the wrong parameters, or refer to an attribute of an object that does not exist or is not of the expected type. Typically dynamic languages work well on relatively small projects with limited complexity. They compile and launch quickly and usually focus on syntax that lends to quick and convenient implementations of common solutions to a lot of standard programming problems. Perl, Javascript, Smalltalk, Lisp, Groovy, and Python are in this camp. Although with Groovy, if you use a clever IDE (Integrated Development Environment) and write straightforward code, the IDE can tell you if your method calls are correct and your references to attributes are accurate. This is because Groovy is a convenience extension on top of Java and Java is a static language.

For me, the most painful thing that can happen in a dynamic language is when you have to refactor the code. For example, if a method is to take a list of a strings, instead of a single string. Generally as large projects grow and mature, they will go through multiple occurrences of refactoring. A refactoring can be quite complex with class definitions being split up into multiple classes, entire function bodies reimplemented using  a completely different approach. I am far from alone in my view of dynamic languages and it is why static languages tend to dominate in large projects and why there are extensions to Javascript, such as TypeScript, that layer a static language on top of a dynamic language.

In a static language you have a significant compile step before code can be launched and run. The static languages that are in my list are Basic (though there are dynamic variants), C, C++, Fortran, and Scala. Static languages also usually oblige the programmer to write more overhead for any code construct. Classes, methods, and attributes have to be more fully declared and there are usually fewer convenience features to minimize the number of lines of code that need to be written. For me, the thing that causes me the most pain in static languages is the thing that gives its virtue, the time it takes to compile the code. Given two different static languages, I will tend to prefer the one with a faster compile time, even if it means that programming in the language is more laborious. Two good examples of this are C vs C++ and Java vs Scala. In C, it is quite difficult to write object oriented programming and in Java it is hard to write multi-threaded programming that glues together or composes function blocks. In C++, if you limit your use of more complex constructions, such as multi-inheritance of multi-parameterized types, the compile time expense is not that much greater. As for Scala, I feel it to be an unusable language based solely on its compile time, though I have never seen or written in a better language when judged solely on the merits of the syntax of a language.

But C++ and Scala come with more hidden costs. Both of them tend towards code that creates many temporary small object allocations and deallocations from memory (the “heap” in Java). And many times, when analyzing the code, it can be difficult to predict the run time speed of a particular block of code and there are many hidden traps. For example, the basic collection type in Scala is the read-only linked list. If you prepend to the list, the cost is just the creation of one new object and a couple of attribute assignments. However, if you append to the linked list, you have to clone the entire list before the append operation can occur, which can rapidly get very expensive. Also, both C++ and Scala allow for aggressive code obfuscation, making it difficult to quickly parse and comprehend code written by another developer. For example, they can give new meanings to operator symbols, have invisible or hidden implicit parameters and, in the case of Scala, seemingly shift the meaning and interpretation of method calls and operators so that they are more like a rules engine than a procedural language. Or to put it another way, with the greater complexity of constructs that both languages allow, it is much easier to write baffling code.

This leads to me to another belief, that I hold strongly though I rarely achieve this in practice, is that the less clever you can make yourself appear when solving a problem, the better the code. When I was in mathematics, the best proofs to a problem, and the ones that were a sign of true genius were elegant, but in retrospect blindingly obvious and simple proofs. The type of proof that made you feel like an idiot for not thinking of it. As an example of this in programming, I give the mark up language HTML. In its first incarnation, it was a ridiculously simple language and compared to other markup languages (such as postscript or other SGML syntaxes) it was primitive and simplistic. It looked like something any programmer with a bit of smarts could have invented. But in the end HTML won out mostly because of its simplicity. Today, nobody would define page content using anything other than HTML. And in some cases, HTML itself is considered too complex and wiki and markdown SGML languages have been developed as being even friendlier and easier to use than HTML.

The problem with C++ and Scala is that they offer seductive ways to write clever code. It is easy to rationalize using complex constructs and libraries to solve relatively simple programming problems. For example in Scala, a simple multi-threaded application will be turned into a complicated messaging application using Akka. Such complex applications can be difficult to troubleshoot when they fail and difficult to comprehend and analyze when trying to extend or alter behavior. But such implementations keep appearing. Code of this type tends to give the programmer that wrote the code an air of smarts and sophistication, a fatal lure. A lure I have felt strongly myself.

Virtual Machine

The other defining difference between languages is whether they run on a virtual machine or not. A virtual machine can prevent memory access violations from crashing your program and provide automatic garbage collection for objects instantiated from the heap. Also, languages written on virtual machines usually have a greater ability to interrogate their own runtime state. For example, if an exception is thrown, it is easy to get at the entire method call stack of the code that caused the error without having to crash the program. Also, virtual machines make it much easier to find and dynamically load new code into the runtime and make it available for programmatic usage, especially when compared to the complexities of using dynamically loadable modules (DLLs on Windows, SOs on other operating systems) when not using a virtual machine.

All the dynamic languages that I have mentioned are implemented on such virtual machines. The only static languages (among the ones that I have mentioned) that run on such virtual machines are Java and Scala (which is written on top of Java). The main competitor to Java is C# (and Google's Go is a recent competitor as well), but I have limited familiarity with C#, so I will not say much about it. However, I believe C# to be as worthy or more worthy a language than Java, but my recent programming career has been focused on linux deploys where C# is not a natural fit.

For me, programming without the safety belt of an underlying virtual machine is madness and should only be contemplated if the requirements of your project demand it, for example if extracting every bit of performance out of the hardware is vital. At the time Java first came out, I was so sold on its virtues that I immediately started doing all my programming in the language only a few months after the very first version  (Java 1.02) was available.

Tool Chains and Libraries

However, there are other considerations when it comes to adopting a particular language. One of them is the number of readily available libraries and the tool chains (such as IDEs) that are built on top of the language. Currently, I believe Python and Java to be the clear winners here, even when compared to Microsoft languages. For example, if you wish to programmatically control robots, you are most likely to find Python or Java to dominate the languages used for the libraries that interface with the robots. Python has the additional win of being pre-installed on almost on all machines not running Microsoft Windows and is becoming the programming language of choice for academic education in programming.

But before declaring Java and Python winners, we should take a moment to look at Javascript. Javascript has one huge advantage over the other languages that cannot be ignored. It is the natural language of web pages, and because of this it also has a strong tool chain built on it and many libraries that can be readily adopted into any project. Also, a large percentage of programmers are familiar and comfortable with Javascript and so naturally tend to desire to use it for other projects besides presenting web pages. Javascript provides one other huge benefit. If you are writing a web application and you write both the server and client in Javascript, you can write end to end unit tests that are executed within a single virtual machine running the Javascript. This is because it is easy to to instantiate and manipulate simulated browsers inside a Javascript program, even if the Javascript program runs entirely as a server side solution. Such an advantage is not to be dismissed lightly.

So, in the end, I believe there are only four real competitors that should be given serious consideration for operating system agnostic languages. They are Python, Javascript, Java, and Groovy. Groovy is only on the list because dynamic languages have their place as facilitators, assemblers, and testers, even in large projects, and Groovy is highly compatible with Java. It is why it is the underlying language for Gradle, a ubiquitous build tool, especially in Android phone development. When Go or Kotlin (Kotlin is another language written on top of Java) reach a certain maturity, they may also be given serious consideration as well.

Java

Over time, Java has become so dominant that it is beginning to pick up a chorus of detractors. Its syntax, when compared to a language like Scala, looks primitive and clunky and Java now has an aura of being an archaic language from a more simple and ignorant era. But even today, Java has strengths not matched by most other languages written on a VM. For example, it is so close to native machine instructions when manipulating arrays of characters, it is possible to build new languages on top of Java that can be compiled and executed with reasonable efficiency. This is something you do not see with Python and Javascript. In general, Java makes a real attempt to use programming constructs that have efficient virtual machine implementations. This allows an experienced programmer to predict with some accuracy the overall speed of execution of a block of code written in Java. 

Java also is a winner against other static languages written on a virtual machine when you look at compile speed. Its simpler syntax and restrictions on locations of files dictated by package membership make it possible to write compilers that can compile a lot of code very quickly. But more than that, if you change code in a particular Java file, the compiler can correctly figure out which other Java files need to be compiled because of the change. Neither Scala or Kotlin do a good job on this, many times recompiling a lot of code even after fairly trivial changes, even if the trivial changes appear to only change the interior logic of method and do not change the signatures of classes or methods. As I have said before, compile speed is one of the primary measuring sticks I use when judging a language, so I see this as a clear and decisive win for Java. TypeScript, the static language extension to Javascript, has similar problems. A javascript language that takes only a few seconds to compile and launch can take minutes to compile when written in TypeScript.

Another reason why Java has gotten a bad reputation is that some of the frameworks written for creating web applications have been painful to use. For example, traditional application server frameworks implementing the full J2EE stack have built up their own set of arcane constructs and have become so burdened with unnecessary complexity that even launching a “hello world” application can take minutes. One of the wins that newer languages provide is that they get to start with simpler and cleaner frameworks for developing applications, one of the main reasons why “node.js”, a Javascript server side implementation, has become so popular. 

But this is a self-inflicted wound done to Java. It is possible to write simpler and more straightforward server side applications in Java. I have done this multiple times and none of the applications I have worked on take more than 60 seconds to compile from scratch and take less than 30 seconds to start up, and some of these applications are fairly sophisticated and complex with hundreds of thousands of lines of code. One of them had over a 100 man years of effort put into it when it reached its most mature state.

So for those who propose a replacement for Java, your new language should have the following characteristics. It must be static with the compiler doing a lot of the work in validating your code, something Scala does extraordinarily well. For example, Scala eliminates code constructs that can cause null pointer exceptions, the bane of Java. The compiler cannot add more than 50% to the overall time of compiling the code when compared to Java, and it must be similarly clever in limiting the amount of compiling done when a few lines of code are changed. When writing code in the language, the execution speed of the implementation should be at least somewhat predictable. To put it another way, the language cannot be too conceptually removed from the underlying hardware that actually executes your program.  The language can be a bit more complex than Java, but not too much so and it should not cater to those who like to write super short method names or use symbols as their method names. In this respect, I am quite impressed with Python which, more than any other language I have encountered, makes it easy to write simple easily comprehensible solutions to simple problems. So, assuming you can make the language static with quick compiles, a Python like language with its heavy emphasis on style, with some of the elegant syntax from Scala (especially for declaring function types) is my best candidate for a language that replaces Java.