-
Notifications
You must be signed in to change notification settings - Fork 9
Npgsql: Use NpgsqlPoint .NET type for marshalling GEO_POINT types. Explore communicating and marshalling GeoJSON types.
#782
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
24f00f6
1897703
297e26d
86d79dc
f433862
da061a9
374d660
2c2c101
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -17,12 +17,9 @@ await Parser.Default.ParseArguments<Options>(args) | |
| { | ||
| var connString = $"Host={options.Host};Port={options.Port};SSL Mode={options.SslMode};" + | ||
| $"Username={options.Username};Password={options.Password};Database={options.Database}"; | ||
| Console.WriteLine($"Connecting to {connString}\n"); | ||
|
|
||
| var dataSourceBuilder = new NpgsqlDataSourceBuilder(connString); | ||
| dataSourceBuilder.EnableDynamicJson(); | ||
| await using var dataSource = dataSourceBuilder.Build(); | ||
| await using var conn = dataSource.OpenConnection(); | ||
| await using var dataSource = GetDataSource(connString); | ||
| var conn = dataSource.OpenConnection(); | ||
|
|
||
| await DatabaseWorkloads.SystemQueryExample(conn); | ||
| await DatabaseWorkloads.BasicConversationExample(conn); | ||
|
|
@@ -34,11 +31,35 @@ await Parser.Default.ParseArguments<Options>(args) | |
| // await dwt.ArrayJsonDocumentExample(); | ||
| await dwt.ObjectPocoExample(); | ||
| await dwt.ArrayPocoExample(); | ||
| await dwt.GeoJsonTypesExample(); | ||
| conn.Close(); | ||
| }); | ||
|
|
||
| } | ||
|
|
||
| public static NpgsqlDataSource GetDataSource(string connString) | ||
| { | ||
| // It's just a testing rig. Do NOT log database credentials in production. | ||
| Console.WriteLine($"Connecting to database: {connString}\n"); | ||
|
|
||
| // Enable JSON POCO mapping and PostGIS/GeoJSON Type Plugin. | ||
| // https://www.npgsql.org/doc/types/json.html | ||
| // https://www.npgsql.org/doc/types/geojson.html | ||
| var dataSourceBuilder = new NpgsqlDataSourceBuilder(connString); | ||
|
|
||
| // Enable JSON POCO mapping Plugin. | ||
| // https://www.npgsql.org/doc/types/json.html | ||
| dataSourceBuilder.EnableDynamicJson(); | ||
|
|
||
| // Enable PostGIS/GeoJSON Type Plugin. | ||
| // https://www.npgsql.org/doc/types/geojson.html | ||
| // `NpgsqlDbType.Geometry` can not be used with CrateDB, | ||
| // see also https://github.com/npgsql/npgsql/issues/2411. | ||
| // dataSourceBuilder.UseGeoJson(); | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Isn't this sending or
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks for your suggestions, here and below. I will investigate them and report back.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I've tried to amend the corresponding INSERT statement, but found that it isn't even executed. The program already fails before, probably based on a type introspection query included in those bunch of statements I've traced using Npgsql introspection SQLSELECT version();
SELECT ns.nspname,
t.oid,
t.typname,
t.typtype,
t.typnotnull,
t.elemtypoid
FROM
(-- Arrays have typtype=b - this subquery identifies them by their typreceive and converts their typtype to a
-- We first do this for the type (innerest-most subquery), and then for its element type
-- This also returns the array element, range subtype and domain base type as elemtypoid
SELECT typ.oid,
typ.typnamespace,
typ.typname,
typ.typtype,
typ.typrelid,
typ.typnotnull,
typ.relkind,
elemtyp.oid AS elemtypoid,
elemtyp.typname AS elemtypname,
elemcls.relkind AS elemrelkind,
CASE
WHEN elemproc.proname='array_recv' THEN 'a'
ELSE elemtyp.typtype
END AS elemtyptype ,
typ.typcategory
FROM
(SELECT typ.oid,
typnamespace,
typname,
typrelid,
typnotnull,
relkind,
typelem AS elemoid,
CASE
WHEN proc.proname='array_recv' THEN 'a'
ELSE typ.typtype
END AS typtype,
CASE
WHEN proc.proname='array_recv' THEN typ.typelem
WHEN typ.typtype='r' THEN rngsubtype
WHEN typ.typtype='m' THEN
(SELECT rngtypid
FROM pg_range
WHERE rngmultitypid = typ.oid)
WHEN typ.typtype='d' THEN typ.typbasetype
END AS elemtypoid ,
typ.typcategory
FROM pg_type AS typ
LEFT JOIN pg_class AS cls ON (cls.oid = typ.typrelid)
LEFT JOIN pg_proc AS proc ON proc.oid = typ.typreceive
LEFT JOIN pg_range ON (pg_range.rngtypid = typ.oid)) AS typ
LEFT JOIN pg_type AS elemtyp ON elemtyp.oid = elemtypoid
LEFT JOIN pg_class AS elemcls ON (elemcls.oid = elemtyp.typrelid)
LEFT JOIN pg_proc AS elemproc ON elemproc.oid = elemtyp.typreceive) AS t
JOIN pg_namespace AS ns ON (ns.oid = typnamespace)
WHERE (typtype IN ('b',
'r',
'm',
'e',
'd')
OR -- Base, range, multirange, enum, domain
(typtype = 'c'
AND relkind='c')
OR -- User-defined free-standing composites (not table composites) by default
(typtype = 'p'
AND typname IN ('record',
'void',
'unknown'))
OR -- Some special supported pseudo-types
(typtype = 'a'
AND (-- Array of...
elemtyptype IN ('b',
'r',
'm',
'e',
'd')
OR -- Array of base, range, multirange, enum, domain
(elemtyptype = 'p'
AND elemtypname IN ('record',
'void'))
OR -- Arrays of special supported pseudo-types
(elemtyptype = 'c'
AND elemrelkind='c')-- Array of user-defined free-standing composites (not table composites) by default
)))
ORDER BY CASE
WHEN typtype IN ('b',
'e',
'p') THEN 0 -- First base types, enums, pseudo-types
WHEN typtype = 'c' THEN 1 -- Composites after (fields loaded later in 2nd pass)
WHEN typtype = 'r' THEN 2 -- Ranges after
WHEN typtype = 'm' THEN 3 -- Multiranges after
WHEN typtype = 'd'
AND elemtyptype <> 'a' THEN 4 -- Domains over non-arrays after
WHEN typtype = 'a' THEN 5 -- Arrays after
WHEN typtype = 'd'
AND elemtyptype = 'a' THEN 6 -- Domains over arrays last
END;
-- Load field definitions for (free-standing) composite types
SELECT typ.oid,
att.attname,
att.atttypid
FROM pg_type AS typ
JOIN pg_namespace AS ns ON (ns.oid = typ.typnamespace)
JOIN pg_class AS cls ON (cls.oid = typ.typrelid)
JOIN pg_attribute AS att ON (att.attrelid = typ.typrelid)
WHERE (typ.typtype = 'c'
AND cls.relkind='c')
AND attnum > 0
AND -- Don't load system attributes
NOT attisdropped
ORDER BY typ.oid,
att.attnum;
-- Load enum fields
SELECT pg_type.oid,
enumlabel
FROM pg_enum
JOIN pg_type ON pg_type.oid=enumtypid
ORDER BY oid,
enumsortorder; |
||
|
|
||
| return dataSourceBuilder.Build(); | ||
| } | ||
|
|
||
| public class Options | ||
| { | ||
| [Option('h', "host", Required = false, HelpText = "Host name to connect to", Default = "localhost")] | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,6 +4,7 @@ | |
| using System.Data; | ||
| using System.Text.Json; | ||
| using System.Threading.Tasks; | ||
| using GeoJSON.Net.Geometry; | ||
| using Newtonsoft.Json; | ||
| using Npgsql; | ||
| using NpgsqlTypes; | ||
|
|
@@ -75,6 +76,7 @@ public async Task CreateTable() | |
|
|
||
| await using (var cmd = new NpgsqlCommand(""" | ||
| CREATE TABLE testdrive.example ( | ||
| id STRING, | ||
| -- Numeric types | ||
| null_integer INT, | ||
| integer INT, | ||
|
|
@@ -166,12 +168,27 @@ INSERT INTO testdrive.example ( | |
| cmd.Parameters.AddWithValue("timestamp_tz", "1970-01-02T00:00:00+01:00"); | ||
| cmd.Parameters.AddWithValue("timestamp_notz", "1970-01-02T00:00:00"); | ||
| cmd.Parameters.AddWithValue("ip", "127.0.0.1"); | ||
|
|
||
| // Container types | ||
| cmd.Parameters.AddWithValue("array", NpgsqlDbType.Json, new List<string>{"foo", "bar"}); | ||
| cmd.Parameters.AddWithValue("object", NpgsqlDbType.Json, new Dictionary<string, string>{{"foo", "bar"}}); | ||
| cmd.Parameters.AddWithValue("geopoint", new List<double>{85.43, 66.23}); | ||
| // TODO: Check if `GEO_SHAPE` types can be represented by real .NET or Npgsql data types. | ||
|
|
||
| // Geospatial types | ||
|
|
||
| // GEO_POINT | ||
| // Alternatively to `NpgsqlPoint`, you can also use `List<double>{85.43, 66.23}`. | ||
| cmd.Parameters.AddWithValue("geopoint", new NpgsqlPoint(85.43, 66.23)); | ||
|
|
||
| // GEO_SHAPE | ||
| // While `GEO_POINT` is transparently marshalled as `NpgsqlPoint`, | ||
| // `GEO_SHAPE` is communicated as scalar `string` type, using WKT or GeoJSON format. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nitpick: CrateDB won't communicate this as Maybe my previously commented workaround by casting the insert to an
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Sorry, it didn't work. |
||
| // TODO: Possibly support transparently converging `GEO_SHAPE` to one of | ||
| // `NpgsqlLSeg`, `NpgsqlBox`, `NpgsqlPath`, `NpgsqlPolygon`, `NpgsqlCircle`. | ||
| cmd.Parameters.AddWithValue("geoshape", "POLYGON ((5 5, 10 5, 10 10, 5 10, 5 5))"); | ||
|
|
||
| // Vector type | ||
| cmd.Parameters.AddWithValue("float_vector", new List<double> {1.1, 2.2, 3.3}); | ||
|
|
||
| cmd.ExecuteNonQuery(); | ||
| } | ||
|
|
||
|
|
@@ -336,6 +353,108 @@ public async Task<List<BasicPoco>> ArrayPocoExample() | |
| } | ||
| } | ||
|
|
||
| public async Task InsertGeoJsonTyped() | ||
| { | ||
| /*** | ||
| * Verify Npgsql PostGIS/GeoJSON Type Plugin with CrateDB. | ||
| * https://www.npgsql.org/doc/types/geojson.html | ||
| * | ||
| * TODO: Does not work yet, because CrateDB communicates GEO_SHAPE as string? | ||
| * The error message is: | ||
|
Comment on lines
+362
to
+363
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. See previous comment about a possible workaround.
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🤷 |
||
| * | ||
| * System.NotSupportedException : The NpgsqlDbType 'Geometry' isn't present in your | ||
| * database. You may need to install an extension or upgrade to a newer version. | ||
| * See also https://github.com/npgsql/npgsql/issues/2411. | ||
| */ | ||
| Console.WriteLine("Running InsertGeoJsonTyped"); | ||
|
|
||
| // Insert single data point. | ||
| await using (var cmd = new NpgsqlCommand(""" | ||
| INSERT INTO testdrive.example ( | ||
| "geoshape" | ||
| ) VALUES ( | ||
| @geoshape | ||
| ); | ||
| """, conn)) | ||
| { | ||
| var point = new Point(new Position(85.43, 66.23)); | ||
| cmd.Parameters.AddWithValue("geoshape", NpgsqlDbType.Geometry, point); | ||
| cmd.ExecuteNonQuery(); | ||
| } | ||
|
|
||
| // Flush data. | ||
| await RefreshTable(); | ||
| } | ||
|
|
||
| public async Task InsertGeoJsonString() | ||
| { | ||
| /*** | ||
| * Communicate GeoJSON types as strings, marshall from/to GeoJSON types manually. | ||
| */ | ||
| Console.WriteLine("Running InsertGeoJsonString"); | ||
|
|
||
| // Insert single data point. | ||
| await using (var cmd = new NpgsqlCommand(""" | ||
| INSERT INTO testdrive.example ( | ||
| "id", | ||
| "geoshape" | ||
| ) VALUES ( | ||
| @id, | ||
| @geoshape | ||
| ); | ||
| """, conn)) | ||
| { | ||
| var point = new Point(new Position(85.43, 66.23)); | ||
| var poly = new Polygon([ | ||
| new LineString([ | ||
| new Position(longitude: 5.0, latitude: 5.0), | ||
| new Position(longitude: 5.0, latitude: 10.0), | ||
| new Position(longitude: 10.0, latitude: 10.0), | ||
| new Position(longitude: 10.0, latitude: 5.0), | ||
| new Position(longitude: 5.0, latitude: 5.0), | ||
| ]) | ||
| ]); | ||
| // TODO: Can GEO_SHAPE types be directly marshalled to a .NET GeoJSON type? | ||
| // Currently, `InsertGeoJsonTyped` does not work yet. | ||
| cmd.Parameters.AddWithValue("id", "point"); | ||
| cmd.Parameters.AddWithValue("geoshape", NpgsqlDbType.Json, JsonConvert.SerializeObject(point)); | ||
| cmd.ExecuteNonQuery(); | ||
|
|
||
| cmd.Parameters.Clear(); | ||
|
|
||
| cmd.Parameters.AddWithValue("id", "polygon"); | ||
| cmd.Parameters.AddWithValue("geoshape", NpgsqlDbType.Json, JsonConvert.SerializeObject(poly)); | ||
| cmd.ExecuteNonQuery(); | ||
| } | ||
|
|
||
| // Flush data. | ||
| await RefreshTable(); | ||
|
|
||
| } | ||
|
|
||
| public async Task<Point?> GeoJsonTypesExample() | ||
| { | ||
| Console.WriteLine("Running GeoJsonTypesExample"); | ||
|
|
||
| // Provision data. | ||
| await CreateTable(); | ||
| // await InsertGeoJsonTyped(); | ||
| await InsertGeoJsonString(); | ||
|
|
||
| // Query back data. | ||
| await using (var cmd = new NpgsqlCommand("SELECT * FROM testdrive.example WHERE id='point'", conn)) | ||
| await using (var reader = cmd.ExecuteReader()) | ||
| { | ||
| reader.Read(); | ||
| // TODO: Can GEO_SHAPE types be directly marshalled to a .NET GeoJSON type? | ||
| // Currently, `InsertGeoJsonTyped` does not work yet. | ||
| var obj = reader.GetFieldValue<JsonDocument>("geoshape"); | ||
| var payload = obj.RootElement.ToString(); | ||
| var geoJsonObject = JsonConvert.DeserializeObject<Point>(payload); | ||
| return (Point?) geoJsonObject; | ||
| } | ||
|
Comment on lines
+445
to
+455
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Check If no matching row exists, await using (var cmd = new NpgsqlCommand("SELECT * FROM testdrive.example WHERE id='point'", conn))
await using (var reader = cmd.ExecuteReader())
{
- reader.Read();
+ if (!reader.Read())
+ {
+ Console.WriteLine("No GeoJSON point record found");
+ return null;
+ }
// TODO: Can GEO_SHAPE types be directly marshalled to a .NET GeoJSON type?
// Currently, `InsertGeoJsonTyped` does not work yet.
var obj = reader.GetFieldValue<JsonDocument>("geoshape");
var payload = obj.RootElement.ToString();
var geoJsonObject = JsonConvert.DeserializeObject<Point>(payload);
- return (Point?) geoJsonObject;
+ return geoJsonObject;
}🤖 Prompt for AI Agents
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That would cause other warnings or more elaborate downstream error handling. We don't want to introduce such details here, because the focus is on concise demo examples. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @amotl, understood! You're right—this aligns with keeping example code concise and focused on demonstrating the core concepts without defensive boilerplate. I have a learning noting that demonstration code in this repository should prioritize clarity over extensive error handling, so I'll keep that in mind for future reviews. Thanks for the clarification! 👍 🧠 Learnings used |
||
| } | ||
|
|
||
| } | ||
|
|
||
| } | ||
Uh oh!
There was an error while loading. Please reload this page.