Using images in a survey¶

This notebook provides sample code for using images with an EDSL survey.

EDSL is an open-source libary for simulating surveys, experiments and other research with AI agents and large language models. Before running the code below, please ensure that you have installed the EDSL library and either activated remote inference from your Coop account or stored API keys for the language models that you want to use with EDSL. Please also see our documentation page for tips and tutorials on getting started using EDSL.

Scenarios¶

A Scenario is a dictionary containing a key/value pair that is used to add data or content to questions in an EDSL survey. Scenarios allow you create variations and versions of questions efficiently, and with data or content from different sources.

EDSL provides a variety of methods for automatically generating scenarios from PDFs, CSVs, docs, tables, lists, dicts -- and images. In the steps below we demonstrate how to create a scenario for an image and use it in a survey.

Learn more about working with scenarios.

Creating a scenario¶

We start by creating a Scenario for an image. For purposes of demonstration, we post a PNG image to the Coop, and then retrieve it as a scenario (this can be done by any user with a Coop account).

Code for posting a PNG file to the Coop (uncomment and replace with your own file):

In [1]:

# from edsl.scenarios.FileStore import PNGFileStore
# fs = PNGFileStore("parrot_logo.png")
# info = fs.push()
# print(info)

Retrieving a Coop file to use as a scenario (replace with the UUID of a desired object):

In [2]:

# from edsl.scenarios.FileStore import PNGFileStore

In [3]:

# png_file = PNGFileStore.pull("e9edda74-d493-42df-92bc-34bf49c6d1bf", expected_parrot_url="https://www.expectedparrot.com")

Converting the retrieved file into a scenario:

In [4]:

from edsl import Scenario

In [5]:

# s = Scenario.from_png(png_file.to_tempfile())

Alternative method for creating a scenario from a local file:

In [6]:

s = Scenario.from_image("parrot_logo.png")

Verify the scenario keys to use in questions:

In [7]:

s.keys()

Out[7]:

['parrot_logo']

Creating questions using the image¶

Next we construct questions with the image scenario. Note that we use a {{ placeholder }} for the scenario key. We also demonstrate how to pipe an answer into a follow-on question:

In [8]:

from edsl import QuestionYesNo, QuestionFreeText, QuestionList, Survey

In [9]:

q1 = QuestionYesNo(
    question_name = "animal",
    question_text = "Is there an animal in this image? {{ parrot_logo }}"
)

In [10]:

q2 = QuestionFreeText(
    question_name = "identify",
    question_text = "Identify the animal in this image: {{ parrot_logo }}"
)

In [11]:

q3 = QuestionList(
    question_name = "colors",
    question_text = "What color(s) is the animal?", 
)

In [12]:

survey = Survey(questions = [q1, q2, q3])

Adding logic and rules for adminitering the questions:

In [13]:

survey = (
    survey
    .add_stop_rule(q1, "animal == 'No'")
    .add_targeted_memory(q3, q2)
)

We use the scenario by adding it to the survey when we run it, as with any other scenarios:

In [14]:

results = survey.by(s).run()

In [15]:

results.select("animal", "identify", "identify_comment", "colors", "colors_comment").print(format="rich")

┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ answer  ┃ answer                    ┃ comment           ┃ answer                    ┃ comment                   ┃
┃ .animal ┃ .identify                 ┃ .identify_comment ┃ .colors                   ┃ .colors_comment           ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ None    │ ["gray", "green",         │                   │ ['gray', 'green',         │ The colors identified are │
│         │ "orange", "pink", "blue", │                   │ 'orange', 'pink', 'blue', │ based on the description  │
│         │ "black"]                  │                   │ 'black']                  │ provided: gray (letter    │
│         │                           │                   │                           │ "E" and brackets), green  │
│         │ The image contains the    │                   │                           │ (bird's body), orange     │
│         │ following colors: gray    │                   │                           │ (bird's beak), pink       │
│         │ (used in the letter "E"   │                   │                           │ (lower part of the bird's │
│         │ and the brackets), green  │                   │                           │ body), blue (lower part   │
│         │ (used in the body of the  │                   │                           │ of the bird's body), and  │
│         │ bird), orange (used in    │                   │                           │ black (bird's eye).       │
│         │ the beak of the bird),    │                   │                           │                           │
│         │ pink (used in the lower   │                   │                           │                           │
│         │ part of the bird's body), │                   │                           │                           │
│         │ blue (used in the lower   │                   │                           │                           │
│         │ part of the bird's body), │                   │                           │                           │
│         │ and black (used in the    │                   │                           │                           │
│         │ eye of the bird).         │                   │                           │                           │
└─────────┴───────────────────────────┴───────────────────┴───────────────────────────┴───────────────────────────┘

In [ ]:

from edsl import Notebook

n = Notebook(path = "image_scenario_example.ipynb")

n.push(description = "Image scenario not working")

from edsl import Notebook

notebook = Notebook.pull("fb6e647b-87e8-4aa4-bae7-f0dfe75d26cc")

# this object is too large to display here