Skip to main content

Discover

The discover method of the Airbyte Protocol returns an AirbyteCatalog: an object which declares all the streams output by a connector and their schemas. It also declares the sync modes supported by the stream (full refresh or incremental). See the beginner's guide to the catalog for more information.

Run a discover command:

poetry run source-survey-monkey-demo discover --config secrets/config.json

The command should succeed, but the schema will be wrong:

{
"type": "CATALOG",
"catalog": {
"streams": [
{
"name": "surveys",
"json_schema": {
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"properties": {
"id": { "type": ["null", "string"] },
"name": { "type": ["null", "string"] },
"signup_date": { "type": ["null", "string"], "format": "date-time" }
}
},
"supported_sync_modes": ["full_refresh"],
"source_defined_primary_key": [["id"]]
}
]
}
}

We'll need to replace the schema surveys.json with a json_schema representation of the records yielded by the stream.

The easiest way to extract the schema from a HTTP response is to use the Connector Builder. You can also paste the schema below, which was generated by the Connector Builder:

{
"$schema": "http://json-schema.org/schema#",
"properties": {
"analyze_url": {
"type": "string"
},
"collect_stats": {
"properties": {
"status": {
"properties": {
"open": {
"type": "number"
}
},
"type": "object"
},
"total_count": {
"type": "number"
},
"type": {
"properties": {
"weblink": {
"type": "number"
}
},
"type": "object"
}
},
"type": "object"
},
"date_created": {
"type": "string"
},
"date_modified": {
"type": "string"
},
"href": {
"type": "string"
},
"id": {
"type": "string"
},
"language": {
"type": "string"
},
"nickname": {
"type": "string"
},
"preview": {
"type": "string"
},
"question_count": {
"type": "number"
},
"response_count": {
"type": "number"
},
"title": {
"type": "string"
}
},
"type": "object"
}
info

If the connector you're building has a dynamic schema, you'll need to overwrite the AbstractSource::streams.


The three connector operations work as expected. You can now upload your connector to your Airbyte instance.In the next section, we'll add the connector to our local Airbyte instance.