Referenced schema by relative URL

[wol-soft]

@wol-soft
I actually get an error as well when trying to evaluate adcp get_products json schema

original https://adcontextprotocol.org/schemas/v2/media-buy/get-products-request.json

simplified:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "/schemas/2.5.0/media-buy/get-products-request.json",
  "title": "Get Products Request",
  "description": "Request parameters for discovering available advertising products",
  "type": "object",
  "properties": {
    "brand_manifest": {
      "$ref": "/schemas/2.5.0/core/brand-manifest-ref.json",
      "description": "Brand information manifest providing brand context, assets, and product catalog. Can be provided inline or as a URL reference to a hosted manifest."
    }
}

brand-manifest-ref.json is this:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "/schemas/2.5.0/core/brand-manifest-ref.json",
  "title": "Brand Manifest Reference",
  "description": "Brand manifest provided either as an inline object or a URL string pointing to a hosted manifest",
  "oneOf": [
    {
      "$ref": "/schemas/2.5.0/core/brand-manifest.json",
      "description": "Inline brand manifest object"
    },
    {
      "type": "string",
      "format": "uri",
      "description": "URL to a hosted brand manifest JSON file. The manifest at this URL must conform to the brand-manifest.json schema."
    }
  ]
}

this can be simplified to

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "/schemas/2.5.0/media-buy/get-products-request.json",
  "title": "Get Products Request",
  "description": "Request parameters for discovering available advertising products",
  "type": "object",
  "properties": {
    "brand_manifest": {
     "oneOf": [
    {
      "$ref": "/schemas/2.5.0/core/brand-manifest.json",
      "description": "Inline brand manifest object"
    },
    {
      "type": "string",
      "format": "uri",
      "description": "URL to a hosted brand manifest JSON file. The manifest at this URL must conform to the brand-manifest.json schema."
    }
  ]
    }
}

which actually builds just fine (praise the lord, your lib is the only one that is able to do so). I wonder though if there is a simple fix without me having to copy paste certain contents around to make it work.

so to recap: it doesnt work the way it is online with reference to a new file

it does work if i copy paste the contents of said file into the property definition.

Originally posted by @LukasGoTom in #57

[wol-soft]

Hi @LukasGoTom,

which error are you facing exactly? I just had a quick look at your simplified structure and resolving the structure works as intended. How do you store the files before trying to generate the schema? Probably the issue is related to the $ref in the schema being relative to the URL the referenced schema is available at and not a valid local reference.

Another issue might be the format validator. The Lib currently has no builtin filters (see the docs) and consequently the generator will fail once he faces the format if you haven't added it on your own.

[LukasGoTom]

@wol-soft prior to posting my issue

1. i actually made sure the folder structure is correct, well should be.
2. I noticed as well the validators and fixed that accordingly before writing my issue.
3. Furthermore your library causes a fatal error when dealing with namespaces with numbers in it, e.g. I believe during the building process you create some classes before creating the final classes. References you use namespaces with the values found in the $ref or $id. If that value is a number, its in invalid php namespace. e.g. you create namespace Schemas\Core\2.5.0\Something which causes a php fatal since Schemas\Core\2 is fatal error for php. I fixed this by sanitizing those values inside the json schemas before generating the clasess

the full error is the nested one, from the other thread. I, wildly, speculate its because in brand-manifest-ref.json there is the same issue as described in 57. The schema is string or object and it doesnt know that its inside an object itself. This is my wild speculation....

i will phrase my speculation with an example:

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "$id": "/schemas/media-buy/get-products-request.json",
  "title": "Get Products Request",
  "description": "Request parameters for discovering available advertising products",
  "type": "object",
  "properties": {
    "brand_manifest": {
      "$ref": "/schemas/core/brand-manifest-ref.json",
      "description": "Brand information manifest providing brand context, assets, and product catalog. Can be provided inline or as a URL reference to a hosted manifest."
    }
  },
  "required": [],
  "additionalProperties": false
}

your generator goes inside brand-manifest-ref.json

it sees this simplified

"oneOf": [
    {
      "$ref": "/schemas/core/brand-manifest.json",
      "description": "Inline brand manifest object"
    },
    {
      "type": "string",
      "format": "uri",
      "description": "URL to a hosted brand manifest JSON file. The manifest at this URL must conform to the brand-manifest.json schema."
    }
  ]

then it is "oh i have the problem from #57 " because it doesnt know that the assignment is inside a property. E.g. brand-manifest-ref.json somehow does not know that the context is the property "brand_manifest" inside an object. It just sees "omg string or object? not sure what to do what that. looked at it in a vaccuum, without context property... i dont know what to generate" this is all wild speculation from my part. maybe you just dont support oneOf in nested schemas?

here is my full error when running my code (inside my phpunit test):

[docker-compose://[/home/lukas/projects/adcp-seller/docker-compose.yml]:app/]:php /app/vendor/phpunit/phpunit/phpunit --configuration /app/phpunit.xml --filter "/(Tests\\Dto\\Generator\\GetProductsGeneratorTest::test_generate_get_products_dtos)( .*)?$/" --test-suffix GetProductsGeneratorTest.php /app/tests/Dto/Generator --teamcity
Testing started at 10:34 ...
PHPUnit 11.5.44 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.28
Configuration: /app/phpunit.xml
Generated class lukastest123\Schemasmediabuygetproductsrequestjson_Itemofarraylogos692d6f4b26b94
Generated class lukastest123\Schemasmediabuygetproductsrequestjson_Colors692d6f4b27492
Generated class lukastest123\Schemasmediabuygetproductsrequestjson_Fonts692d6f4b275bf
Generated class lukastest123\Schemasmediabuygetproductsrequestjson_Metadata692d6f4b279f2
Generated class lukastest123\Schemasmediabuygetproductsrequestjson_Itemofarrayassets692d6f4b276af
Generated class lukastest123\Schemasmediabuygetproductsrequestjson_Product_catalog692d6f4b27ae8
Generated class lukastest123\Schemasmediabuygetproductsrequestjson_Itemofarraydisclaimers692d6f4b27c1e
Generated class lukastest123\Schemasmediabuygetproductsrequestjson_Contact692d6f4b27d38
Generated class lukastest123\Schemasmediabuygetproductsrequestjson_Metadata692d6f4b27dd0
Generated class lukastest123\Schemasmediabuygetproductsrequestjson_schemascorebrandmanifestjson
Generated class lukastest123\Schemasmediabuygetproductsrequestjson
Duplicated signature a5f4f71ba6fee18929a9eaaa1428ec33 for class Schemascorebrandmanifestrefjson_schemascorebrandmanifestjson. Redirecting to Schemasmediabuygetproductsrequestjson_schemascorebrandmanifestjson

PHPModelGenerator\Exception\SchemaException: No nested schema for composed property Schemascorebrandmanifestrefjson in file /app/tests/Dto/Generator/downloadedSchemas2/schemas/v2/media-buy/schemas/core/brand-manifest-ref.json found
/app/vendor/wol-soft/php-json-schema-model-generator/src/PropertyProcessor/Property/BaseProcessor.php:300
/app/vendor/wol-soft/php-json-schema-model-generator/src/Utils/ResolvableTrait.php:17
/app/vendor/wol-soft/php-json-schema-model-generator/src/PropertyProcessor/Property/BaseProcessor.php:298
/app/vendor/wol-soft/php-json-schema-model-generator/src/PropertyProcessor/Property/BaseProcessor.php:71
/app/vendor/wol-soft/php-json-schema-model-generator/src/PropertyProcessor/PropertyFactory.php:54
/app/vendor/wol-soft/php-json-schema-model-generator/src/SchemaProcessor/SchemaProcessor.php:139
/app/vendor/wol-soft/php-json-schema-model-generator/src/SchemaProcessor/SchemaProcessor.php:100
/app/vendor/wol-soft/php-json-schema-model-generator/src/SchemaProcessor/SchemaProcessor.php:67
/app/vendor/wol-soft/php-json-schema-model-generator/src/ModelGenerator.php:115
/app/tests/Dto/Generator/GetProductsGeneratorTest.php:57

Time: 00:00.046, Memory: 10.00 MB

There was 1 error:

1) Tests\Dto\Generator\GetProductsGeneratorTest::test_generate_get_products_dtos
PHPModelGenerator\Exception\SchemaException: No nested schema for composed property Schemascorebrandmanifestrefjson in file /app/tests/Dto/Generator/downloadedSchemas2/schemas/v2/media-buy/schemas/core/brand-manifest-ref.json found

/app/vendor/wol-soft/php-json-schema-model-generator/src/PropertyProcessor/Property/BaseProcessor.php:300
/app/vendor/wol-soft/php-json-schema-model-generator/src/Utils/ResolvableTrait.php:17
/app/vendor/wol-soft/php-json-schema-model-generator/src/PropertyProcessor/Property/BaseProcessor.php:298
/app/vendor/wol-soft/php-json-schema-model-generator/src/PropertyProcessor/Property/BaseProcessor.php:71
/app/vendor/wol-soft/php-json-schema-model-generator/src/PropertyProcessor/PropertyFactory.php:54
/app/vendor/wol-soft/php-json-schema-model-generator/src/SchemaProcessor/SchemaProcessor.php:139
/app/vendor/wol-soft/php-json-schema-model-generator/src/SchemaProcessor/SchemaProcessor.php:100
/app/vendor/wol-soft/php-json-schema-model-generator/src/SchemaProcessor/SchemaProcessor.php:67
/app/vendor/wol-soft/php-json-schema-model-generator/src/ModelGenerator.php:115
/app/tests/Dto/Generator/GetProductsGeneratorTest.php:57

ERRORS!
Tests: 1, Assertions: 0, Errors: 1.

Process finished with exit code 2

this error goes away, if I, manually copy paste brand-manifest-ref.json into get-products-request.json AND delete brand-manifest-ref.json because otherwise, it tries to build it again because it finds it in the folder

[wol-soft]

Ok, I think I understood your setup now. Thanks for the detailled explanation.

To me it looks like you've put all schemas into one directory (the base schema get-products-request.json as well as the referenced schemas brand-manifest-ref.json and brand-manifest.json) - not the exact same directory but a subdirectory. Then you use RecursiveDirectoryProvider to loop over all files in the directory and tries to generate a model from the json files found in the directory. During this process, when visiting the brand-manifest-ref.json the model generator fails as the composition can't be used as a root schema to generate an object as the second branch of the oneOf composition is just a plain string.

Instead, your setup from a directory perspective should be more like the original structure:

schemas/
├─ media-buy/
│  ├─ get-products-request.json
│  ├─ …
└─ core/
   ├─ brand-manifest-ref.json
   ├─ brand-manifest.json

and your RecursiveDirectoryProvider points to schemas/media-buy. With this setup, the brand-manifest-ref.json will not be converted standalone into an object but only used as the reference from the generation process for get-products-request.json.

[LukasGoTom]

oh thanks a lot, I will try this out tomorrow, even though... I'm not sure how...

The reason with the weird subdirectories is that I didnt want to rewrite all $ref and $id entries... because you somehow open them relatively to the current file. The issue is here afaik:

\PHPModelGenerator\Model\SchemaDefinition\SchemaDefinitionDictionary::parseExternalFile

you do

        $jsonSchemaFilePath = filter_var($jsonSchemaFile, FILTER_VALIDATE_URL)
            ? $jsonSchemaFile
            : $this->sourceDirectory . '/' . $jsonSchemaFile;

/schemas/core/brand-manifest.json is not a valid url right, so you do

$this->sourceDirectory . '/' . $jsonSchemaFile

this will be relative because sourceDirectory will be

for example

/schemas/media-buy

and jsonSchemaFile will be
/schemas/core/brand-manifest.json

so it will look for

/schemas/media-buy/schemas/core/brand-manifest.json

if

/schemas/media-buy/schemas/core/brand-manifest.json has a different reference, say /schemas/core/brand.json, it will look for

/schemas/media-buy/schemas/core/schemas/core/brand.json

its quite weird but i believe thats what it does

Now actually rethinking it i could fix it with:

1. a custom implementation of \PHPModelGenerator\Model\SchemaDefinition\SchemaDefinitionDictionary::parseExternalFile
2. I skip brand-manifest-rel.json and others in the iterator, would be custom implementation as well
3. another weird folder structure, say I put the main json file here: /my-main-file/get-products-request.json
   a. all other files I put into /my-sub-files/...
   b. I rewrite all refs and ids to match the new structure

[wol-soft]

The issue here is, that the $refs only use a relative URL and not an absolute URL and consequently, as you wrote, the resolver does not know where to find the file and looks it up locally. Probably an alternative approach could be, to only download the main schema files (in this case get-products-request.json) and then, before processing the files, prefixing all $ref with the domain to ensure it's a valid URL which passes the filter_var and is loaded afterwards. This would also reduce the files you have to set up locally.

The replacement can be done via sed, a PHP script before or something else or you could extend the RecursiveDirectoryProvider and extend the getSchemas function to rewrite all $refs

[wol-soft]

Probably a decorator like the one below could be helpful:

class SchemaProviderRefDecorator implements SchemaProviderInterface
{
    public function __construct(private SchemaProviderInterface $schemaProvider) {}

    public function getSchemas(): iterable
    {
        foreach ($this->schemaProvider->getSchemas() as $schema) {
            yield $schema->withJson($this->patchURL($schema->getJson()));
        }
    }

    public function getBaseDirectory(): string
    {
        return $this->schemaProvider->getBaseDirectory();
    }

    private function patchURL(array $schema): array
    {
        foreach ($schema as $key => &$value) {
            if ($key === '$ref' && is_string($value)) {
                $value = 'https://adcontextprotocol.org' . $value;
            }

            if (is_array($value)) {
                $value = $this->patchURL($value);
            }
        }

        return $schema;
    }
}

And then, as suggested above, only the main schemas are stored locally and processed by the RecursiveDirectoryProvider which is then passed to the decorator:

$generator
    ->generateModelDirectory(__DIR__ . '/result')
    ->generateModels(
        new SchemaProviderRefDecorator(new RecursiveDirectoryProvider(__DIR__ . '/schema')),
        __DIR__ . '/result',
    );

One issue I see is, that files which are loaded dynamically from the URLs referenced in the main schemas would also need the decoration as they reference again without full URLs

[LukasGoTom]

@wol-soft thanks for your input, I really appreciate it. I fear the prefix approach... isnt so easy. I dont only have to prefix the domain, I, somewhen, have to patch the paths as well since you generate a namespace out of them if I'm not very mistaken

your decorator does that "before everything" which works for URL, but not for fixing the namespace bugs.

e.g.

https://adcontextprotocol.org/schemas/2.5.0/core/brand-manifest.json will be bugged since it tries to generate a namespace Schemas\2.5.0\ out of it somewhere in the chain of actions. Others too https://adcontextprotocol.org/schemas/marketing-experts/brand-manifest.json would be a bug as well since
\Schemas\Marketing-experts is not a valid php namespace either.

i will come up with some idea... currently leaning towards relative path adjustments e.g. I download it in a way the namespace bugs are gone, and in a way that the root folder is has no invalid schemas in it, by referencing sub schemas with ../ or so. I'll let you know what direction i finally swinged

[wol-soft]

The namespace issue definitely needs a fix in the library. I've added a patch for that in 0fc7d6b

[wol-soft]

Hi @LukasGoTom,

additionally to the namespace fix, I've started to work on some changes to improve the $ref resolving to support more setups in #95. This will need some work but in the end it should support cloning the https://github.com/adcontextprotocol/adcp repository and then building from there the https://github.com/adcontextprotocol/adcp/tree/main/static/schemas/source/media-buy directory with correct resolving of the context-absolute paths to the schemas located in the core directory without manual tweaks.

A schema provider for the hosted version is not easily implementable as the directories like https://adcontextprotocol.org/schemas/2.5.0/media-buy/ don't return a list of available schemas. Otherwise an option would be to implement a RecursiveHostedSchemasProvider which would fetch all schemas and pass them to the generator, similar to the local RecursiveDirectoryProvider.

[LukasGoTom]

@wol-soft love it. I notice as well that they dont have the version number in the git hosted version. I actually wrote myself something similar to "RecursiveHostedSchemasProvider" but since I needed to patch multiple things, I wrote it into a separate "DownloaderService" class

[wol-soft]

Just out of curiosity: How does your provider know which schemas exist?

[LukasGoTom]

@wol-soft this is my class, it downloads AND does some sanitization to folder and file structure... I think it would break with recursion. e.g. say we have a "product" schema. the product itself has "relatedProduct" that is just another "product" schema. I believe the folder structure would break

use Webmozart\Assert\Assert;


/** Wol soft schema generator expects a specific folder structure which we will just give to him
 * Alternative would be to fix the implementation via fork, problem lies somewhere here @see \PHPModelGenerator\Model\SchemaDefinition\SchemaDefinitionDictionary::parseExternalFile
 * */
class SchemaDownloaderService
{
    public const string REGEXP_VERSION_NR_MATCH = '#^\d+(?:\.\d+)*(?:-[A-Za-z0-9]+)?$#';
    private string $baseUrl;
    private string $localRoot;
    private array $downloaded = [];

    public function __construct(string $baseUrl, string $localRoot)
    {
        $this->baseUrl = rtrim($baseUrl, '/');
        $this->localRoot = rtrim($localRoot, '/');
    }

    public function download(string $schemaPathWithFilename, string $currentRelativePath = ''): void
    {
        // Normalize absolute $ref starting with '/'
        $schemaPathWithFilename = '/' . ltrim($schemaPathWithFilename, '/');


        $nextRelativePath = $this->sanitizeFileAndFolderNames(dirname($currentRelativePath . $schemaPathWithFilename));
        $localFile = $this->sanitizeFileAndFolderNames($this->localRoot. $currentRelativePath . $schemaPathWithFilename);
        // Avoid double download
        if (isset($this->downloaded[$localFile])) {
            return;
        }
        $this->downloaded[$localFile] = true;

        $url = $this->baseUrl . $schemaPathWithFilename;

        $currentLocalDirectory = dirname($localFile);
        if (!mkdir($currentLocalDirectory, 0777, true) && !is_dir($currentLocalDirectory)) {
            throw new \RuntimeException(sprintf('Directory "%s" was not created', $currentLocalDirectory));
        }

        $content = file_get_contents($url);
        Assert::notFalse($content, "Failed to download $url");

        $putReturn = file_put_contents($localFile, $content);
        Assert::notFalse($putReturn, "Failed to file put to " . $localFile);

        // Parse JSON and recurse into nested refs
        $json = json_decode($content, true, 512, JSON_THROW_ON_ERROR);

        if (!is_array($json)) {
            return;
        }

        $this->scanRefs($json, $nextRelativePath);
    }

    private function scanRefs(array $data, string $currentRelativePath = ''): void
    {
        foreach ($data as $key => $value) {
            if ($key === '$ref' && is_string($value) && str_starts_with($value, '/')) {
                $this->download($value, $currentRelativePath);
            }

            if (is_array($value)) {
                $this->scanRefs($value, $currentRelativePath);
            }
        }
    }

    private function sanitizeFileAndFolderNames(string $path): string
    {
        $parts = explode(DIRECTORY_SEPARATOR, $path);

        // Remove parts that look like a version number (e.g., 2.5.0, 1.2, 3.1.4-beta)
        $filteredParts = array_filter($parts, static function ($part) {
            return !preg_match(self::REGEXP_VERSION_NR_MATCH, $part);
        });

        $filteredParts = array_map(static function($part) {
            return str_replace('-', '', $part);
        }, $filteredParts);

        return implode(DIRECTORY_SEPARATOR, $filteredParts);
    }
}

[wol-soft]

With f698d45 I got a successful build of the media-buy folder from the cloned adcp repository. I exposed the internals of file-loading in the parseExternalFile function via the SchemaProviderInterface. With this change, it's possible to create a decorator for the RecursiveDirectoryProvider:

<?php

declare(strict_types=1);

namespace PHPModelGenerator\SchemaProvider;

use PHPModelGenerator\Model\SchemaDefinition\JsonSchema;

class SchemaProviderRefDecorator implements SchemaProviderInterface
{
    public function __construct(private SchemaProviderInterface $schemaProvider) {}

    public function getSchemas(): iterable
    {
        return $this->schemaProvider->getSchemas();
    }

    public function getBaseDirectory(): string
    {
        return $this->schemaProvider->getBaseDirectory();
    }

    public function getRef(string $currentFile, ?string $id, string $ref): JsonSchema
    {
        if (str_starts_with($ref, '/schemas')) {
            $ref = str_replace('/schemas', '/schemas/source', $ref);
        }

        return $this->schemaProvider->getRef($currentFile, $id, $ref);
    }
}

This decorator can then be applied:

$generator = new ModelGenerator(
    (new GeneratorConfiguration())
        ->addFormat('uri', new AlwaysHappyValidator())
        ->addFormat('date-time', new AlwaysHappyValidator())
        ->addFormat('email', new AlwaysHappyValidator())
        ->addFormat('date', new AlwaysHappyValidator())
        ->addFormat('jpg', new AlwaysHappyValidator())
        ->addFormat('mp4', new AlwaysHappyValidator())
        ->setNamespacePrefix('\\ADCP\\MediaBuy')
        ->setImmutable(false),
);

$generator
    ->generateModelDirectory(__DIR__ . '/result')
    ->generateModels(
        new SchemaProviderRefDecorator(
            new RecursiveDirectoryProvider(__DIR__ . '/../../adcp/static/schemas/source/media-buy')
        ),
        __DIR__ . '/result',
    );

You can also try the improveRefResolving branch with this addition for your setup. The branch includes also the fixes for stable code output, namespace naming and the recursion error 😉

It should also be possible to use the getRef to setup and load from the correct URL instead of the cloned repository if you must rely on the hosted version of the schemas.

Before merging this addition, I need to add some more tests and additional documentation.