Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for namespaces #12

Merged
merged 5 commits into from
May 10, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 46 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
# Upstash Vector Go Client

[![Go Reference](https://pkg.go.dev/badge/github.com/upstash/vector-go.svg)](https://pkg.go.dev/github.com/upstash/vector-go)

> [!NOTE]
> **This project is in GA Stage.**
>
> The Upstash Professional Support fully covers this project. It receives regular updates, and bug fixes. The Upstash team is committed to maintaining and improving its functionality.
> The Upstash Professional Support fully covers this project. It receives regular updates, and bug fixes.
> The Upstash team is committed to maintaining and improving its functionality.

This is the Go client for [Upstash](https://upstash.com/) Vector.

Expand All @@ -19,10 +22,12 @@ go get github.com/upstash/vector-go

## Usage

In order to use this client, head out to [Upstash Console](https://console.upstash.com) and create a vector database.

### Initializing the client

There are two pieces of configuration required to use the Upstash Vector index client: an REST token and REST URL.
Find your configuration values in the console dashboard at [https://console.upstash.com/](https://console.upstash.com/).
The REST token and REST URL configurations are required to initialize an Upstash Vector index client.
Find your configuration values in the console dashboard at [Upstash Console](https://console.upstash.com/).

```go
import (
Expand Down Expand Up @@ -80,14 +85,31 @@ func main() {

Upstash vector indexes support operations for working with vector data using operations such as upsert, query, fetch, and delete.

Index operations are associated with a namespace variable.
You can specify a namespace for an index client with `Namespace(ns string)` function.
If namespace is not specified, default namespace is used.

```go
import (
"github.com/upstash/vector-go"
)

func main() {
index := vector.NewIndex("<UPSTASH_VECTOR_REST_URL>", "<UPSTASH_VECTOR_REST_TOKEN>")

// Returns a new index client associated with namespace "<NAMESPACE>"
ns := index.Namespace("<NAMESPACE>")
}
```

### Upserting Vectors

All vectors upserted to index must have the same dimensions.

Upsert can be used to insert new vectors into index or to update
existing vectors.

#### Upsert many
#### Upsert Many

```go
upserts := []vector.Upsert{
Expand Down Expand Up @@ -287,3 +309,23 @@ err := index.Reset()
```go
info, err := index.Info()
```

### List Namespaces

All the names of active namespaces can be listed.

```go
namespaces, err := index.ListNamespaces()
for _, ns : range namespaces {
fmt.Println(ns)
}
```

### Delete Namespaces

A namespace can be deleted entirely if it exists.
The default namespaces cannot be deleted.

```go
err := index.DeleteNamespace("ns")
```
14 changes: 7 additions & 7 deletions delete.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@ package vector

const deletePath = "/delete"

// Delete deletes the vector with the given id and reports
// whether the vector is deleted. If a vector with the given
// id is not found, Delete returns false.
// Delete deletes the vector in a namespace with the given id and reports whether the vector is deleted.
// If a vector with the given id is not found, Delete returns false.
// If namespace is not specified, the default namespace is used.
func (ix *Index) Delete(id string) (ok bool, err error) {
data, err := ix.sendBytes(deletePath, []byte(id))
data, err := ix.sendBytes(deletePath, []byte(id), true)
if err != nil {
return
}
Expand All @@ -16,10 +16,10 @@ func (ix *Index) Delete(id string) (ok bool, err error) {
return
}

// DeleteMany deletes the vectors with the given ids and reports
// how many of them are deleted.
// DeleteMany deletes the vectors in a namespace with the given ids and reports how many of them are deleted.
// If namespace is not specified, the default namespace is used.
func (ix *Index) DeleteMany(ids []string) (count int, err error) {
data, err := ix.sendJson(deletePath, ids)
data, err := ix.sendJson(deletePath, ids, true)
if err != nil {
return
}
Expand Down
91 changes: 49 additions & 42 deletions delete_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,49 +15,56 @@ func randomString() string {
}

func TestDelete(t *testing.T) {
client, err := newTestClient()
require.NoError(t, err)

id := randomString()
err = client.Upsert(Upsert{
Id: id,
Vector: []float32{0, 1},
})
require.NoError(t, err)

t.Run("existing id", func(t *testing.T) {
ok, err := client.Delete(id)
require.NoError(t, err)
require.True(t, ok)
})

t.Run("non existing id", func(t *testing.T) {
ok, err := client.Delete(randomString())
require.NoError(t, err)
require.False(t, ok)
})
for _, ns := range namespaces {
t.Run("namespace_"+ns, func(t *testing.T) {
client, err := newTestClientWithNamespace(ns)
require.NoError(t, err)
id := randomString()
err = client.Upsert(Upsert{
Id: id,
Vector: []float32{0, 1},
})
require.NoError(t, err)

t.Run("existing id", func(t *testing.T) {
ok, err := client.Delete(id)
require.NoError(t, err)
require.True(t, ok)
})

t.Run("non existing id", func(t *testing.T) {
ok, err := client.Delete(randomString())
require.NoError(t, err)
require.False(t, ok)
})
})
}
}

func TestDeleteMany(t *testing.T) {
client, err := newTestClient()
require.NoError(t, err)

id0 := randomString()
id1 := randomString()
id2 := randomString()
err = client.UpsertMany([]Upsert{
{
Id: id0,
Vector: []float32{0, 1},
},
{
Id: id1,
Vector: []float32{5, 10},
},
})
require.NoError(t, err)

count, err := client.DeleteMany([]string{id0, id1, id2})
require.NoError(t, err)
require.Equal(t, 2, count)
for _, ns := range namespaces {
t.Run("namespace "+ns, func(t *testing.T) {
client, err := newTestClientWithNamespace(ns)
require.NoError(t, err)

id0 := randomString()
id1 := randomString()
id2 := randomString()
err = client.UpsertMany([]Upsert{
{
Id: id0,
Vector: []float32{0, 1},
},
{
Id: id1,
Vector: []float32{5, 10},
},
})
require.NoError(t, err)

count, err := client.DeleteMany([]string{id0, id1, id2})
require.NoError(t, err)
require.Equal(t, 2, count)
})
}
}
141 changes: 72 additions & 69 deletions embedding_test.go
Original file line number Diff line number Diff line change
@@ -1,84 +1,87 @@
package vector

import (
"github.com/stretchr/testify/require"
"testing"
"time"

"github.com/stretchr/testify/require"
)

func TestEmbedding(t *testing.T) {
client, err := newEmbeddingTestClient()
require.NoError(t, err)
for _, ns := range namespaces {
t.Run("namespace_"+ns, func(t *testing.T) {
client, err := newEmbeddingTestClient(ns)
require.NoError(t, err)

id0 := "tr"
id1 := "jp"
id2 := "uk"
id3 := "fr"
err = client.UpsertDataMany([]UpsertData{
{
Id: id0,
Data: "Capital of Türkiye is Ankara.",
Metadata: map[string]any{"country": id0, "capital": "Ankara"},
},
{
Id: id1,
Data: "Capital of Japan is Tokyo.",
Metadata: map[string]any{"country": id1, "capital": "Tokyo"},
},
{
Id: id2,
Data: "Capital of England is London.",
Metadata: map[string]any{"country": id2, "capital": "London"},
},
{
Id: id3,
Data: "Capital of France is Paris.",
Metadata: map[string]any{"country": id3, "capital": "Paris"},
},
})
require.NoError(t, err)
id0 := "tr"
id1 := "jp"
id2 := "uk"
id3 := "fr"
err = client.UpsertDataMany([]UpsertData{
{
Id: id0,
Data: "Capital of Türkiye is Ankara.",
Metadata: map[string]any{"country": id0, "capital": "Ankara"},
},
{
Id: id1,
Data: "Capital of Japan is Tokyo.",
Metadata: map[string]any{"country": id1, "capital": "Tokyo"},
},
{
Id: id2,
Data: "Capital of England is London.",
Metadata: map[string]any{"country": id2, "capital": "London"},
},
{
Id: id3,
Data: "Capital of France is Paris.",
Metadata: map[string]any{"country": id3, "capital": "Paris"},
},
})
require.NoError(t, err)

require.Eventually(t, func() bool {
info, err := client.Info()
require.NoError(t, err)
return info.PendingVectorCount == 0
}, 10*time.Second, 1*time.Second)
require.Eventually(t, func() bool {
info, err := client.Info()
require.NoError(t, err)
return info.PendingVectorCount == 0
}, 10*time.Second, 1*time.Second)

t.Run("score", func(t *testing.T) {
scores, err := client.QueryData(QueryData{
Data: "where is the capital of Japan?",
TopK: 1,
})
require.NoError(t, err)
require.Equal(t, 1, len(scores))
require.Equal(t, id1, scores[0].Id)
})
t.Run("score", func(t *testing.T) {
scores, err := client.QueryData(QueryData{
Data: "where is the capital of Japan?",
TopK: 1,
})
require.NoError(t, err)
require.Equal(t, 1, len(scores))
require.Equal(t, id1, scores[0].Id)
})

t.Run("with metadata", func(t *testing.T) {
scores, err := client.QueryData(QueryData{
Data: "Which country's capital is Ankara?",
TopK: 1,
IncludeMetadata: true,
})
require.NoError(t, err)
require.Equal(t, 1, len(scores))
require.Equal(t, id0, scores[0].Id)
require.Equal(t, map[string]any{"country": "tr", "capital": "Ankara"}, scores[0].Metadata)
})
t.Run("with metadata", func(t *testing.T) {
scores, err := client.QueryData(QueryData{
Data: "Which country's capital is Ankara?",
TopK: 1,
IncludeMetadata: true,
})
require.NoError(t, err)
require.Equal(t, 1, len(scores))
require.Equal(t, id0, scores[0].Id)
require.Equal(t, map[string]any{"country": "tr", "capital": "Ankara"}, scores[0].Metadata)
})

t.Run("with metadata filtering", func(t *testing.T) {
query := QueryData{
Data: "Where is the capital of France?",
TopK: 1,
IncludeMetadata: true,
Filter: `country = 'fr'`,
}
t.Run("with metadata filtering", func(t *testing.T) {
query := QueryData{
Data: "Where is the capital of France?",
TopK: 1,
IncludeMetadata: true,
Filter: `country = 'fr'`,
}

scores, err := client.QueryData(query)
require.NoError(t, err)
require.Equal(t, 1, len(scores))
require.Equal(t, id3, scores[0].Id)
require.Equal(t, map[string]any{"country": "fr", "capital": "Paris"}, scores[0].Metadata)
})
scores, err := client.QueryData(query)
require.NoError(t, err)
require.Equal(t, 1, len(scores))
require.Equal(t, id3, scores[0].Id)
require.Equal(t, map[string]any{"country": "fr", "capital": "Paris"}, scores[0].Metadata)
})
})
}
}
10 changes: 5 additions & 5 deletions fetch.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@ package vector

const fetchPath = "/fetch"

// Fetch fetches one or more vectors with the ids passed into f.
// When f.IncludeVectors is true, values of the vectors are also
// returned. When f.IncludeMetadata is true, metadata of the vectors
// are also returned, if any.
// Fetch fetches one or more vectors in a namespace with the ids passed into f.
// If IncludeVectors is set to true, the vector values are also returned.
// If IncludeMetadata is set to true, any associated metadata of the vectors is also returned, if any.
// If namespace is not specified, the default namespace is used.
func (ix *Index) Fetch(f Fetch) (vectors []Vector, err error) {
data, err := ix.sendJson(fetchPath, f)
data, err := ix.sendJson(fetchPath, f, true)
if err != nil {
return
}
Expand Down
Loading
Loading