0.2.0

Merged PRs

doltgresql

• 67: Potentially fix release workflow
  GH Actions cannot bypass branch rules, so this is an attempt to have them play by the same rules.
• 63: Fixes for issues: 40, 42, 43
  This contains fixes for:
  • #40
  • #42
  • #43
    In particular, the Getting Started example was a good test, as it actually uncovered quite a few issues separate from the example. These have all been fixed. I know you're not going to like how we're handling TableFuncExprs (we weren't before), but it's way easier than going to the GMS side and adding functionality for now. With some of these bugs fixed, I'd actually expect a decent bump in the sqllogictests TBH.
• 62: Added parser tests
  This PR adds a lot of parser tests (all validated against Postgres 15), in addition to fixing a ton of issues with the synopses. Never again will I rely on the Postgres documentation being correct, in some ways it's even worse than MySQL's documentation. They should have just posted the grammar, because it looks handwritten and it's actually wrong in a lot of places. Rant aside, there's actually a bit of non-test stuff here, but it's all to support the test generation, so this probably doesn't need more than a glance over.
  Some fun statistics:

  Test	Percentage	Number	Description
  Total	100%	332547	The total number of tests that were added
  Unimplemented	96.86%	322092	The number of tests that do not parse at all
  Parses	2.09%	6936	The number of tests that parse, but cannot be converted to a Vitess AST
  Converts To AST	1.05%	3524	The number of tests that can successfully create a Vitess AST (accuracy not tested)

• 61: Added the ability to get an exact permutation from parser tests
  This allows you to pass in a number and get that exact permutation. So that means that, even though SELECT has 144 septillion permutations, if we give it 25,757,684,286,468,754, then it will give us exactly that permutation. So to generate a set of tests, we just generate some amount of random numbers between [0, permutations), and we have our tests!
• 60: Implemented support for various expressions needed by sqllogictest
• 59: Added permutation counting, variable definitions to parser test generation
  This adds the ability to count the number of permutations without having to actually loop through them all and increment one by one. This also adds variable declaration support (the where VAR is one of: in the synopsis). With those two, I can now verify that with some of the variables defined in SELECT, we're up to 144 septillion variations lol. I also removed the repetition strategy, which cut out a ton of tests that probably weren't necessary (exponential on top of combinatorial).
  This is a prelude to the next PR. I've found a strategy to be able to grab a deterministic set of tests (we can randomly decide the set, but given the proper input it's deterministic), but it requires knowing the number of possible permutations ahead of time, hence this PR.
• 58: Parser Tests Tool
  This adds a tool that can download synopses from the Postgres documentation page, save those synopses under the testing/generation/command_docs/synopses folder, iterate over each synopsis to create a StatementGenerator that can create all permutations of a query based on the synopsis, check that the queries are valid against a Postgres installation, and then create a test file verifying the level of implementation for the query in DoltgreSQL (whether the query parses or can be fully transformed). The idea here is that we run this once (or whenever we upgrade to a newer Postgres version, like version 16), and then as we continue developing, we'll naturally add support for more statements. This tracks regressions too, so if something used to successfully parse and now it doesn't, then we can see that we've broken something. At least, that's the idea.
  The major issue is that the permutation count is massive. The tool even supports variables as defined by each synopsis, and without adding those, I got to over 50 billion permutations for just SELECT before I stopped it. We need to find a way to get some reasonable subset for the massive statements, so we can discuss that later.
  Regarding this PR though, the tool is mostly complete. There are a few "issues"
  • Doesn't currently check with an actual Postgres installation yet. This is easy to add, but I didn't bother since we need to find a way to get the total number of tests down first.
  • Doesn't actually do anything with variables yet. Permutation counts exhibit exponential behavior, so I wouldn't be surprised if the final numbers are in the quintillions.
  • Downloaded synopses require manual fiddling. This is mostly due to the Postgres documentation being inconsistent with whitespace, and writing a tool to deal with these inconsistencies was too much. Not all files require fiddling (variable substitution is handled by the tool), and it's not a lot of fiddling either, but it's there.
    Besides those things, this is fully complete. The single example file in testing/generation/command_docs/output/abort_test.go works from start to finish (downloading synopsis, generating queries, creating test file), so you can see what it'll look like once we figure out how to get a subset.
    Regarding how to review this, I'd mainly say to just like at that test file. You can also check out some of the synopses to see how some of them differ from what's on the website. If you really want to, you can look at the parser and generator, but it's not required.
• 57: Better type handling in harness, support COUNT(*) expressions and CAST
• 56: Null literal handling
  Also adds unit tests for error handling during extended query execution, and some debug logging for messages received.
• 55: Fixed INT8 handling, support for testing statements, proper connection error handling
  This primarily implements four things:
  1. We now discard all messages until we reach a Sync message when an error occurs, which is consistent with the documentation.
  2. Proper support for boolean results. The sqllogictests make use of them, and GMS doesn't have a native boolean type, so we treat all booleans as Int32 types. You can read more in the comments.
  3. Proper error handling is the multi-message test case. Our conn.Query calls in tests only return specific kinds of errors, and all other errors are put into the returned rows. This differs from conn.Exec, which returns all errors. Previously, we never checked for an error in rows (since I never knew an error would be put there), so it was getting swallowed. We now properly handle those errors.
  4. Expanded statement support for Describe messages. I noticed in the zachmu/err-handling-repro branch that the test file testing/go/prepared_statement_test.go is inconsistent with the general test framework, as it runs the setup statements using conn.Query rather than conn.Execute. We don't support Describe at all right now, as GMS currently requires analyzing a statement to determine the output schema, and analyzing some statements produces execution side effects. As a workaround, we start a transaction, execute the query, record the schema, and rollback the transaction. This does not work for any commands that implicitly commit the transaction, so this adds another hack to fake it by running an equivalent DROP to select CREATE statements. This does cause an implicit commit, but it's better than failing for now.
• 53: Support arbitrary message size from clients
  This change implements support for arbitrary messages sizes from clients. Previously we were limited to 2k messages.
  Now that we are reading a message of known size, there's no need for Receive() to return more than a single message: we always fetch one at a time from the connection.
  Also refactors the main listening loop into manageable chunks.
• 52: Added Format CI and Release CD
  The formatter does not auto-format and push to the repo, it just fails for now. The release scripts are untested, but they seem like they should work. They're built off of Dolt's scripts, but since there are quite a few differences, I may have missed something that's required. I'm fairly confident that it's all included and supposed to work though. We'll know for sure when we attempt to use it.
• 51: Warn on non-ascending index
  This is to bootstrap compatibility until we can implement descending order indexes. Since GMS assumes all indexes are ordered ascending, this will not impact correctness, just produce slightly different schemas.
• 50: sqllogictest runner
  First attempt at sqllogictest runner
  Also fixes several small bugs related to prepared statement caching, types, and tuples, and updates to the latest gms / vitess.
• 49: Added full Postgres AST to Vitess AST conversion
  This completely implements the conversion from the Postgres AST to the Vitess AST. All parser statements are accounted for, so from here we just need to test and flesh out the remaining incomplete implementations.
• 48: Fix misspelt Postgres
• 45: Added replication limitation to README
• 38: Removed CGO dependency
  This breaks parser compatibility with a few spatial types, however those would fail at the engine layer anyway, so it's fine to be broken at the parser layer for now. Removing the dependency on CGO lets us cross-compile.

Closed Issues

• 43: No databases on macOS arm64?
• 42: pgfutter to load csv gives: driver bad connection. db.Ping() not supported?
• 40: Dolt Table functions not suppoorted yet
• 44: Document backup and replication needs/compability