NAV Navbar
cURL/Example

Introduction

Welcome to Hivemind!

The first step in order to start using Hivemind is to get an account. You can do this by reaching out to us using the Contact Us section of our website. Once you're all setup, you'll be creating tasks like a PRO in no time.

This documentation is there to help you get started and creating your first tasks. For further help, please reach out to support@hvmd.io. We strive to improve this documentation too, so please send feedback and suggesstions there too. Happy Tasking!

Getting Started

Authenticating

Hivemind uses API keys to allow access to the API. You can obtain your API key from the 'Settings' area in Studio.

Hivemind expects every call to the API to be accompanied by an API key in the 'Authorization' HTTP header in the following format:

Authorization: ApiKey API_MYAPIKEY1234

curl "api_endpoint_here" -H "Authorization: ApiKey API_MYAPIKEY1234"

Obtaining your endpoint address

The endpoint address you use to access the Hivemind API depends on which instance of Studio you use.

Typically, your endpoint will be of the form:

https://<YOUR_STUDIO_ADDRESS>/api

E.g. https://studio.gamma.hvmd.io/api

Roles and Permissions

Hivemind supports multiple types of user, each with their own set of capabilities across Studio and Workbench. The following table summarises them.

Hivemind Roles

Creating a Task

You can use Studio to create and preview a task before you send it to your workers to complete.

Alternatively, you can use the following endpoint to create it via the API:

POST /api/tasks

Attributes

Attribute Detail Example Optional?
name The name of the task to be displayed to the worker. My First Task N
projectId The id of the project this task is grouped under. 1 N
ownerId The id of the user that owns the task. Defaults to authenticated user if omitted. 2 Y
documentation A Markdown text field displayed to each worker outlining what to do. Y
outputSchema A JSON Schema field defining the desired shape of the data output. We utilise the excellent react-jsonschema-form library to render JSON into forms. {} N
targetAgreement The desired consensus from the completed work. If omitted, on-platform agreement will be disabled and the following two fields must be set to the same value. 0.51 Y
minimumIterations The minimum number of workers to send each instance to. 2 N
maximumIterations The maximum number of workers to send each instance to before an instance is marked as 'Complete' 4 N
instanceTemplate A Handlebars template for instance instructions in Markdown format. Using Instance Templates Y
instanceSchemaTemplate A Handlebars template for instance override schemas in JSON format. Y
agreementOptions An object used to specify Advanced Agreement Options for specifying field-by-field agreement checking. AgreementOptions Y
options Additional options used to specify how the form is rendered. Options Y
errors A list of error-cases to give the workers the option to flag issues to the task creator ["Website down", "File missing", "No data available"] Y

Adding Documentation

Documentation in Hivemind is created using Markdown fields. We use the CommonMark specification, with several enhancements of our own, outlined below.

Displaying Images, Video and linking to Files

You can use the following syntax in your Markdown to render a link to a file, or embed an image or video uploaded to a Task:

Link to file:

![file, filename.ext]

Image:

![image, image.png]

Video:

![video, video.mp4]

Using Coloured Text

{color:#63dd26}This is a coloured piece of text.{color:lightblue} Nesting colours {color} is okay.{color}

results in:

This is a coloured piece of text. Nesting colours is okay.

Data Quality

Data quality can be achieved on a Hivemind task by setting the targetAgreement field on a task. By default, this will enforce a full equality mode on the results where an iteration is said to agree with another if the results submitted by a worker are identical to anothers.

This can be a little too strict for some cases where you may want to account for differences in case, spacing or look for general similarity in values submitted. In this instance, you can use the Advanced Agreement Options to set data quality parameters on a field-by-field basis. The best way to get an idea of how to use the advanced mode and AgreementOptions is to create a task in Studio and explore the 'Data Quality' tab.

Offline Agreement Checking

Using the API, you can also setup agreement criteria using custom code in your language of choice. You can disable platform agreement checking by setting targetAgreement to null on the task. In this case, minimumIterations must be set equal to maximumIterations.

You can download results from the platform using the following endpoint:

GET /api/tasks/{taskId}/results

or

GET /api/tasks/{taskId}/instances/{instanceId}/results

In cases where you are unable to reach consensus, you can use the following endpoint to have another worker complete the instance:

GET /api/tasks/{taskId}/instances/{instanceId}/reiterate

We have provided a workflow wrapper called Hivemind+ to connect a Hivemind task with a local SQL database using Python.

Defining Output

{
  "title": "A Form Example",
  "type": "object",
  "definitions":{
    "cities" : {
      "type": "integer",
      "enum": [1,2,3],
      "enumNames": ["London", "New York", "Oxford"],
      "title": "City of Residence"
    }
  },
  "required": [
    "firstName",
    "lastName",
    "city"
  ],
  "properties": {
    "firstName": {
      "type": "string",
      "title": "First name"
    },
    "lastName": {
      "type": "string",
      "title": "Last name"
    },
    "city": {
      "$ref" : "#/definitions/cities"
    }
  }
}

The JSON on the right results in a form being rendered with 3 mandatory fields: firstName, lastName and city.

A Form Example

Uploading files

curl -F �file=@path/to/local/file� -H "Authorization: ApiKey API_MYAPIKEY1234" https://studio.{yours}.hvmd.io/api/tasks/{id}/files

Files can be uploaded for use by a task using the follow API endpoint:

POST /api/tasks/{taskId}/files

Files can also be uploaded in Hivemind Studio from the Files tab in the task creation/edit screen.

Files are unique by filename within an instance. Uploading a file with the same name to the same task will overwrite the previously uploaded file.

Once you have uploaded files, you can embed/link to them in your markdown.

Using Instance Templates

Instance templates can be used to set the format of the instance instructions using Markdown and then have specific 'Data' from the instances injected into it.

Instance data:

{  
   "Data":{  
      "URL":"https://hvmd.io",
      "Name":"Hivemind",
      "LastVisited":"2 July 2018"
   },
   "Collect":"the author of the most recent blog post"
}

For example, the following instance template:

Go to the following URL: [{{Data.Name}}]({{Data.URL}}) and find {{Collect}}.

When used with the instance data on the right, will result in the following instance instruction:

Go to the following URL:Hivemindand find the author of the most recent blog post.

As you can see, the Data.LastVisited field on the instance is not used in the template and will be safely ignored.

Adding Instances

Instances can be added to a task using the following endpoint:

POST /api/tasks/{taskId}/instances

Attribute Detail Example Optional?
name A name for this instance that is displayed in Studio, but not to the contributor. My Awesome Instance N
instructions A Markdown field to contain the instructions for the worker. Is also a Handlebars template and if set, will override the instance template from the task. Y
overrideSchema A JSON field that will be injected into the Tasks outputSchema into the definitions block. Schema Overrides Y
data A JSON field that is injected into the various Handlebars templates on the Task and Instance. { "URL" : "https://hvmd.io" } Y
tags A string[] to be used for storing any data you want to keep with the instance for later processing. ["id:7", "database:customer"] Y

Schema Overrides

For example, the following outputSchema

{
  "title": "An Overridden Form Example",
  "type": "object",
  "definitions":{
    "cities" : {
      "type": "integer",
      "enum": [1,2,3],
      "enumNames": ["London", "New York", "Oxford"],
      "title": "City of Residence"
    }
  },
  "properties": {
    "city": {
      "$ref" : "#/definitions/cities"
    }
  }
}

And the following overrideSchema

{
  "cities" : {
      "type": "integer",
      "enum": [4,5,6],
      "enumNames": ["Bangkok", "Los Angeles", "Amsterdam"],
      "title": "City of Residence"
    }
}

You can customize the output form on a per-instance basis by using the overrideSchema field on each instance.

For example, if you want to change the available values in a dropdown on each instance, you can define that field in the definitions block of the task outputSchema and redefine it in the overrideSchema of the instance.

Using an overrideSchema similar to the one on the right results in the form being rendered with updated dropdown values for 'City of Residence':

An Overridden Form Example

Using an Instance Schema Template

You can use an instanceSchemaTemplate on a task to provide a Handlebars template for the overrideSchema.

The example on the right achieves the same result as the previous example, but uses the templating functionality.

For the following Data field on an instance:


{
  "cities" : ["Bangkok", "Los Angeles", "Amsterdam"],
  "values" : [4, 5, 6]
}

And the following instanceSchemaTemplate on the task

{
  "cities": {
      "type": "integer",
      "enum" : [
          {{#each values}}
                    "{{this}}" {{#if @last}}{{else}},{{/if}}
                {{/each}}
      ],
      "enumNames": [
            {{#each cities}}
                    "{{this}}" {{#if @last}}{{else}},{{/if}}
                {{/each}}
      ],
      "title": "City of Residence"
    }
}

Using Schedules

Using the Schedules feature of Hivemind, you can periodically generate new instances for a task based on a templated set of tasks.

Studio provides a nice interface by which to do this when you create a task, so we'll describe how to do it with the API here.

In order to create a schedule for task, you can use the following endpoint:

POST /api/tasks/{taskId}/schedules

Attribute Detail Example Optional?
name A unique string representing the schedule Daily @ 3pm N
cronExpressions An array of cronExpressions to define the frequency that the schedule is run. ["0 15 * * *"] N
timezone An IANA timezone identifier indicating which timezone to use. If omitted, UTC is used. Europe/London Y
disabled A flag indicating this schedule should be temporarily disabled. true Y

Adding instances to a Schedule

Once you have created a schedule, you can add instances to it using the following endpoint:

POST /api/tasks/{taskId}/schedules/{scheduleId}/instances

The format of the instance is identical to adding instances to a task, including the use of Handlebars templates for instructions and schemas. For details, please see here

When a Schedule is executed according to the cron expression(s), any instances on the schedule are created on the task as if they were created on the task by yourself. You can view a log of the previous executions on a schedule by examining the following endpoint:

GET /api/tasks/{taskId}/schedules/{scheduleId}/history

Understanding the lifecycle of a Schedule

Determining when a schedule is run is largely dependent on the status of the task. Furthermore, some changes in the status of a task can have an impact on the schedules. Please note the following when using the Scheduler feature:

Qualifications

Qualifications in Hivemind are used to control which workers have the ability to complete a task. A worker must have all qualifications assigned to a task in order to see the task and complete instances.

If an Instance has qualifications assigned to it, then the worker must have all qualifications assigned to the parent Task AND Instance in order to complete that work. Using this mechanism, it is possible to granularly control who completes the work.

You use the following endpoint to get a list of all the qualifications in Hivemind available to you:

GET /api/qualifications

While the following endpoint will allow you to create new qualifications:

POST /api/qualifications

curl -X POST -H "Authorization: ApiKey API_MYAPIKEY" https://{STUDIO}/api/qualifications -d '{ "name" : "HUMAN" }'
Attribute Detail Example Optional?
name A unique string representing the qualification HUMAN N

Adding Qualifications to Tasks, Instances and Users

Once created, the following endpoint can be used to assign the qualification to a Task:

POST /api/tasks/{taskId}/qualifications/{qualificationId}

Instance:

POST /api/tasks/{taskId}/instances/{instanceId}/qualifications/{qualificationId}

And Users:

POST /api/users/{userId}/qualifications/{qualificationId}

Sending a HTTP DELETE to these endpoints will reverse the action and remove the qualification.

Agents

An Agent in Hivemind is our way of representing a piece of machine intelligence. Agents are represented as a type of User and are given work to do in the same way (i.e. through Qualifications). They have a great deal of flexibility on how they complete work, like doing the work using a particular AI algorithm or even sending the work out to external platforms in order to harness other workforces.

Some of the Agents we've implemented are outlined below. In order to use them, we recommend using the tiles in Studio as they are customised to help you tailor your task for the particular agents.

Extracting text from images using OCR

Hivemind utilises the popular Tesseract library to provide OCR features to Hivemind. The agent supports a variety of image formats and can extract text in a number of languages.

Extracting entities from text

The Entity Extraction agent allows you to send bodies of text into Hivemind and extract entities.

Using the Mechanical Turk platform through Hivemind

Using the MTurk Agent, Hivemind allows you to harness the power 000's of workers through the Mechanical Turk crowdworking platform.

Using our implementation, you are able to define a task EXACTLY as you would for Hivemind, try it out in Workbench and then send it to MTurk using the AGENT-MTURK qualification.

Tasks are presented to MTurkers in exactly the same way it is for work done through Workbench and results are returned in the same way.

Getting started with MTurk

The first step will be to obtain an Access ID and Secret Key from your MTurk account. Instructions from Amazon are outlined here.

Once you have these keys, you can enter them into Hivemind in the settings section in Studio. For security, it is advisable that the user represented by these credentials only have access to the mechanicalturk:* area of AWS.

After you have entered these details, you will be able to see the balance of your MTurk account in the Studio settings - this is a good indication of whether the setup has been successful.

You are now ready to create a task in Hivemind for MTurk. Once you add the qualification AGENT-MTURK to a task (or use the template in Studio), you will be able to set the Reward on the task, which is the amount in USD that is to be paid to worker excluding Amazon's fees for using the platform.

Controlling quality of work in MTurk

Invariably the work completed on the MTurk platform is sometimes completed by bots or other automated entities looking to abuse the system. In cases like this, you can utilise the following Hivemind mechanism to reject the work and reiterate the instance.

DELETE /api/tasks/{taskId}/instances/{instanceId}/iterations/{iterationId}

This will result in the work being rejected on MTurk and the worker not being paid. It will also automatically create a new iteration and send it out to MTurk for completion by another worker.

Pause on an MTurk Task

Amazon MTurk does not support Pause functionality like you can do on a pure Hivemind task. In the instance where you Pause an MTurk task, this will simply put a hold on further instances being sent to MTurk, until you Unpause the task.

While a task is paused, you may still get results back from MTurk as workers complete any instances that have already been sent.

Glossary

AgreementOptions

For the following outputSchema

{
  "title": "A Form Example",
  "type": "object",
  "definitions":{
    "names": {
      "type": "object",
      "properties": {
        "first": {
          "type": "string",
          "title": "First name"
        },
        "last": {
          "type": "string",
          "title": "Last name"
        },
        "middle": {
          "type": "array",
          "items": {
            "type": "string"
          },
          "title": "Middle name(s)"
        }
      },
      "required": [
        "first",
        "last"
      ]
    },
    "cities" : {
      "type": "integer",
      "enum": [1,2,3],
      "enumNames": ["London", "New York", "Oxford"],
      "title": "City of Residence"
    }
  },
  "required": [
    "age",
    "city"
  ],
  "properties": {
    "names": {
      "$ref": "#/definitions/names"
    },
    "age" : {
      "type" : "number",
      "title" : "Age"
    },
    "city": {
      "$ref" : "#/definitions/cities"
    }
  }
}

The agreementOptions may look like:

{
  "options" : {
    "names":{
      "options": {
        "first": {
          "options":{
            "ignoreCase" : true,
            "ignoreWhitespace" : true,
            "algorithm" : "jarowinkler",
            "requiredDistance" : 0.15
          }
        },
        "last" : {
          "options": {
            "ignore" : true
          }
        },
        "middle": {
          "options": {
            "ignoreOrder": true,
            "items": {
              "options": {
                "ignoreCase": true,
                "ignoreWhitespace": true,
                "ignoreSymbols": true
              }
            }
          }
        }     
      }
    },
    "age" : {
      "options": {
        "decimalPlaces" : 0,
        "errorConstant" : 1
      }
    },
    "city" : {
      "options": {
        "ignore" : false
      }
    }
  }
}
Attribute Detail Example Optional?
options Field-by-field options that define critera for advanced agreement checking Y

AgreementOptions - Options

Options available for advanced agreement checking are dependent on the type of the field. All types support the below option. Additional options supported are outlined below by type.

Attribute Detail Example Optional?
ignore Ignore agreement for this field true/false Y

string

For string fields, the following additional options can be passed.

Attribute Detail Example Optional?
ignoreCase Don't take case into account when comparing values true/false Y
ignoreWhitespace Ignore white spaces when comparing values wherever they occur true/false Y
ignoreSymbols Ignore symbols (E.g. #, @, �) when comparing values true/false Y
algorithm Which algorithm to use when comparing values. Can be equality (default), jarowinkler or levenshtein jarowinkler Y
requiredDistance The required distance threshold when algorithm is set to jarowinkler (default: 0.1) or levenshtein (default: 5) 0.7 Y

number

For number fields, the following additional options can be passed.

Attribute Detail Example Optional?
ignoreSign Don't take sign (+/-) into account when comparing values true/false Y
decimalPlaces Number of decimal places to take into account when comparing values. E.g. 3.145 ~ 3.152 when decimalPlaces = 2. Mid-point values round away from zero, meaning -1.5 ~ -2 when decimalPlaces = 0 2 Y
errorConstant The absolute value difference allowed between two values. E.g. 98 ~ 100 when errorConstant >= 2 2 Y
errorFactor The proportional value difference allowed between two values as a percentage of the larger value. E.g. 98 ~ 100 when errorFactor >= 0.02 0.02 Y

boolean

There are currently no additional options for boolean fields.

array

For array fields, the following additional options can be passed. Note the special items option, which can be used to pass agreement options for the elements of the array.

Attribute Detail Example Optional?
ignoreOrder Don't take order into account when comparing arrays. Arrays will agree if there is any ordering that gives pairwise agreement true/false Y
ignoreNullOrEmpty Ignore elements of an array that are empty strings, arrays or objects true/false Y
items The agreement options to use when comparing the array elements {"options":{...}} Y

object

For objects, an optional agreement options value can be provided for each field within the object using the field key.

Attribute Detail Example Optional?
foo The agreement options to use when comparing the "foo" field {"options":{...}} Y
bar The agreement options to use when comparing the "bar" field {"options":{...}} Y

Options

Attribute Detail Example Optional?
formOptions Options that effect the way the form is rendered FormOptions Y

Options FormOptions

Attribute Detail Example Optional?
fullWidthFields A string[] of fields that should be rendered using the entire width of the form ["firstName", "lastName"] Y