Creating a memory persistence component

Key takeaways

Memory persistence Create a memory persistence component and perform CRUD operations.

Introduction

In this tutorial, you will learn how to create a persistence component, which will store some objects in memory. Then, we will see how to perform CRUD operations, such as adding data, reading it, updating stored values and deleting them. We will use a dummy object, which has the characteristic of being identifiable via an id parameter. All concepts learned here can be expanded to other more complex objects.

Create a memory persistence component

In order to create our memory persistence component, we will follow these two steps.

Step 1 - Creating a dummy class

We will create a dummy class, which represents an object that is identifiable via an id. PIP.Services provides us with the IStringdentifiable interface that can be used to create data objects with this characteristic. We will also define a content parameter for the class, which can include any text.

Once we created our class, we will create three instances of it, each with a different id. For one of the objects, we will use None to let the program define its id. The code will look something like this:

See: IStringdentifiable

import { IStringIdentifiable } from 'pip-services3-commons-nodex';

export class Dummy implements IStringIdentifiable {
    public id: string;
    public key: string;
    public content: string;
}

let dummy1: Dummy = {id:'1', key:'key 1', content:'content 1'};
let dummy2: Dummy = { id: 'id 1', key: "key 2", content: "Content 1" };
let dummy3: Dummy = {id: null, key: 'key 3', content:'content 3'};

See: IStringdentifiable

using PipServices3.Commons.Data;
using System.Runtime.Serialization;

[DataContract]
public class Dummy : IStringIdentifiable
{
    [DataMember(Name = "id")]
    public string Id { get; set; }

    [DataMember(Name = "key")]
    public string Key { get; set; }

    [DataMember(Name = "content")]
    public string Content { get; set; }
}

var dummy1 = new Dummy { Id = "1", Key = "key 1", Content = "content 1" };
var dummy2 = new Dummy { Id = "id 1", Key = "key 2", Content = "Content 1" };
var dummy3 = new Dummy { Id = null, Key = "key 3", Content = "content 3" };

See: IStringdentifiable

package persistence

type Dummy struct {
	Id      string `json:"id"`
	Key     string `json:"key"`
	Content string `json:"content"`
}


dummy1 := Dummy("1", "key 1", "content 1")
dummy2 := Dummy("id 1", "key 2", "content 2")
dummy3 := Dummy(nil, "key 3", "content 3")

See: IStringdentifiable

import 'package:pip_services3_commons/pip_services3_commons.dart';

class Dummy implements IStringIdentifiable, ICloneable {
  @override
  String? id;
  String? key;
  String? content;

  Dummy({String? id, String? key, String? content})
      : id = id,
        key = key,
        content = content;

  Map<String, dynamic> toJson() {
    return <String, dynamic>{'id': id, 'key': key, 'content': content};
  }

  void fromJson(Map<String, dynamic> json) {
    id = json['id'];
    key = json['key'];
    content = json['content'];
  }

  @override
  Dummy clone() {
    return Dummy(id: id, key: key, content: content);
  }
}

var dummy1 = Dummy(id: '1', key: 'key 1', content: 'content 1');
var dummy2 = Dummy(id: 'id 1', key: 'key 2', content: 'Content 1');
var dummy3 = Dummy(id: null, key: 'key 3', content: 'content 3');


from pip_services3_commons.data import IStringIdentifiable

class Dummy(IStringIdentifiable): 
    def __init__(self, id: str = None, key: str = None, content: str = None): 
        self.id = id 
        self.key = key 
        self.content = content 
        
dummy1 = Dummy('1', 'key 1', 'content 1') 
dummy2 = Dummy('id 1', 'key 2', 'content 2') 
dummy3 = Dummy(None, 'key 3', 'content 3')
Not available

Step 2 – Create a memory persistence object

The next step is to create a memory persistence object. Here, we need to use the IdentifiableMemoryPersistence class, which is an abstract persistence component that stores data in memory and implements CRUD operations over data items with unique ids. We will also define two methods namely, getPageByFilter and getOneByKey, which will be used to read the persisted values.

See: IdentifiableMemoryPersistence

import { FilterParams } from 'pip-services3-commons-nodex';
import { PagingParams } from 'pip-services3-commons-nodex';
import { DataPage } from 'pip-services3-commons-nodex';
import { IdentifiableMemoryPersistence } from 'pip-services3-data-nodex';



export class MyMemoryPersistence
    extends IdentifiableMemoryPersistence<Dummy, string> {


    public constructor() {
        super();
    }

    private composeFilter(filter: FilterParams): (item: Dummy) => boolean {
        filter = filter != null ? filter : new FilterParams();
        let id = filter.getAsNullableString("id");
        let tempIds = filter.getAsNullableString("ids");
        let ids = tempIds != null ? tempIds.split(',') : null;
        let key = filter.getAsNullableString("key");

        return (item: Dummy) => {
            if (id != null && item.id != id)
                return false;
            if (ids != null && ids.indexOf(item.id) < 0)
                return false;
            if (key != null && item.key != key)
                return false;
            return true;
        };
    }

    public async getPageByFilter(correlationId: string, filter: FilterParams, paging: PagingParams): Promise<DataPage<Dummy>> {
        return await super.getPageByFilter(correlationId, this.composeFilter(filter), paging, null, null);
    }

    public async getOneById(correlationId: string, key: string): Promise<Dummy> {
        for(let item of this._items){
            if (item.key == key) {
                this._logger.trace(correlationId, "Found object by key=" + key);
                return item;
            }
        }

        this._logger.trace(correlationId, "Cannot find by key=" + key);
    }
}

...

let persistence = new MyMemoryPersistence();

See: IdentifiableMemoryPersistence

using PipServices3.Commons.Data;
using PipServices3.Data.Persistence;

using System;
using System.Collections.Generic;
using System.Runtime.Serialization;
using System.Threading.Tasks;

namespace ExampleApp
{

    public class MyMemoryPersistence : IdentifiableMemoryPersistence<Dummy, string>
    {
        public MyMemoryPersistence() : base()
        {
            _maxPageSize = 1000;
        }

        private List<Func<Dummy, bool>> ComposeFilter(FilterParams filter)
        {
            filter ??= new FilterParams();

            var id = filter.GetAsNullableString("id");
            var key = filter.GetAsNullableString("key");
            var ids = filter.GetAsNullableString("ids");
            var idsList = ids != null ? new List<string>(ids.Split(',')) : null;

            return new List<Func<Dummy, bool>>()
            {
                (item) =>
                {
                    if (id != null && item.Id != id)
                        return false;
                    if (key != null && item.Key != key)
                        return false;
                    if (idsList != null && idsList.IndexOf(item.Id) < 0)
                        return false;
                    return true;
                }
            };
        }

        public async Task<Dummy> GetOneByKeyAsync(string correlationId, string key)
        {
            Dummy item = null;

            lock (_lock)
            {
                item = _items.Find((dummy) => { return dummy.Key == key; });
            }

            if (item != null) _logger.Trace(correlationId, "Found object by key={0}", key);
            else _logger.Trace(correlationId, "Cannot find by key={0}", key);

            return item;
        }

        public async Task<DataPage<Dummy>> GetPageByFilterAsync(string correlationId, FilterParams filter, PagingParams paging)
        {
            return await base.GetPageByFilterAsync(correlationId, this.ComposeFilter(filter), paging);
        }
    }
}

...

var persistence = new MyMemoryPersistence();

See: IdentifiableMemoryPersistence

// extends IdentifiableMemoryPersistence
import (
	"context"
	"strings"

	cdata "github.com/pip-services3-gox/pip-services3-commons-gox/data"
	cpersist "github.com/pip-services3-gox/pip-services3-data-gox/persistence"
)

type Dummy struct {
	Id      string `json:"id"`
	Key     string `json:"key"`
	Content string `json:"content"`
}

type MyMemoryPersistence struct {
	*cpersist.IdentifiableMemoryPersistence[Dummy, string]
}

func NewMyMemoryPersistence() *MyMemoryPersistence {
	return &MyMemoryPersistence{IdentifiableMemoryPersistence: cpersist.NewIdentifiableMemoryPersistence[Dummy, string]()}
}

func composeFilter(filter *cdata.FilterParams) func(item Dummy) bool {
	if filter == nil {
		filter = cdata.NewFilterParams(make(map[string]string))
	}

	id, _ := filter.GetAsNullableString("id")
	temp_ids, idsOk := filter.GetAsNullableString("ids")

	var ids *[]string
	if idsOk {
		*ids = strings.Split(temp_ids, ",")

	}

	key, _ := filter.GetAsNullableString("key")

	return func(dummy Dummy) bool {
		if id != "" && dummy.Id != id {
			return false
		}
		if key != "" && dummy.Key != key {
			return false
		}

		if len(*ids) > 0 {
			for _, v := range *ids {
				if dummy.Id == v {
					return true
				}
			}
			return false
		}
		return true
	}
}

func (c *MyMemoryPersistence) GetOneByKey(ctx context.Context, correlationId string, key string) (item Dummy, err error) {
	for _, val := range c.Items {
		if val.Key == key {
			item = val
			break
		}
	}
	return item, err
}

func (c *MyMemoryPersistence) GetPageByFilter(ctx context.Context, correlationId string, filter *cdata.FilterParams, paging *cdata.PagingParams) (page DataPage[Dummy], err error) {

	if &filter == nil {
		filter = cdata.NewEmptyFilterParams()
	}

	tempPage, err := c.IdentifiableMemoryPersistence.GetPageByFilter(ctx, correlationId, composeFilter(filter), *paging, nil, nil)

	// Convert to DummyPage
	dataLen := int64(len(tempPage.Data)) // For full release tempPage and delete this by GC
	data := make([]Dummy, dataLen)
	for i, v := range tempPage.Data {
		data[i] = v
	}
	page = NewDummyPage(&dataLen, data)
	return page, err
}

// ...

persistence := NewMyMemoryPersistence()

See: IdentifiableMemoryPersistence

import 'package:pip_services3_commons/pip_services3_commons.dart';
import 'package:pip_services3_data/pip_services3_data.dart';

class MyMemoryPersistence extends IdentifiableMemoryPersistence<Dummy, String> {
  MyMemoryPersistence() : super();

  bool Function(Dummy item) _composeFilter(FilterParams? filter) {
    filter = filter ?? FilterParams();
    var id = filter.getAsNullableString('id');
    var tempIds = filter.getAsNullableString('ids');
    var ids = tempIds != null ? tempIds.split(',') : null;
    var key = filter.getAsNullableString('key');

    return (item) {
      if (id != null && item.id != id) {
        return false;
      }
      if (ids != null && !ids.contains(item.id)) {
        return false;
      }
      if (key != null && item.key != key) {
        return false;
      }
      return true;
    };
  }

  Future<DataPage<Dummy>> getPageByFilter(
      String? correlationId, FilterParams? filter, PagingParams? paging) async {
    return await super.getPageByFilterEx(
        correlationId, _composeFilter(filter), paging, null, null);
  }

  @override
  Future<Dummy?> getOneById(String? correlationId, String key) async {
    for (var item in items) {
      if (item.key == key) {
        logger.trace(correlationId, 'Found object by key=' + key);
        return item;
      }
    }

    logger.trace(correlationId, 'Cannot find by key=' + key);
  }
}

...

var persistence = MyMemoryPersistence();

See: IdentifiableMemoryPersistence


from typing import Callable, Optional, Any 
from pip_services3_data.persistence.IdentifiableMemoryPersistence import IdentifiableMemoryPersistence 
from pip_services3_commons.data import FilterParams, PagingParams, DataPage 

class MyMemoryPersistence(IdentifiableMemoryPersistence): 
    def __init__(self): 
        super(MyMemoryPersistence, self).__init__() 
 
    def __compose_filter(self, filter_params: FilterParams) -> Callable[[Dummy], bool]: 
        filter_params = filter_params or FilterParams() 
        id = filter_params.get_as_nullable_string("id") 
        temp_ids = filter_params.get_as_nullable_string("ids") 
        ids = temp_ids.split(",") if temp_ids is not None else None 
        key = filter_params.get_as_nullable_string("key") 
 
        def inner(item: Dummy) -> bool: 
            if id is not None and item.id != id: 
                return False 
            if ids is not None and item.id in ids: 
                return False 
            if key is not None and item.key != key: 
                return False 
            return True 
 
        return inner 
 
    def get_page_by_filter(self, correlation_id: Optional[str], filter: Any, paging: PagingParams, sort: Any = None, 
                           select: Any = None) -> DataPage: 
        return super().get_page_by_filter(correlation_id, self.__compose_filter(filter), paging, sort, select) 
 
    def get_one_by_key(self, correlation_id, key): 
        for item in self._items: 
            if item.key == key: 
                self._logger.trace(correlation_id, "Found object by key={}", key) 
                return item 
             
        self._logger.trace(correlation_id, "Cannot find by key={}", key) 
 

persistence = MyMemoryPersistence()
Not available

CRUD operations

Now that we have a persistence object, we will perform CRUD operations.

Create the persisted objects

To add values to the persistence object, we will use the create method. This method asks for two parameters: correlation_id and the object to persist. For the correlation_id we will use None as in our example we are not interested in following a sequence of operations.

let result = await persistence.create(null, dummy1);
console.log('Created Dummy with ID: ' + result.id);

result = await persistence.create(null, dummy2);
console.log('Created Dummy with ID: ' + result.id);

result = await persistence.create(null, dummy3);
console.log('Created Dummy with ID: ' + result.id);


var result = await persistence.CreateAsync(null, dummy1);
Console.WriteLine($"Created Dummy with ID: {result.Id}");

result = await persistence.CreateAsync(null, dummy2);
Console.WriteLine($"Created Dummy with ID: {result.Id}");

result = await persistence.CreateAsync(null, dummy3);
Console.WriteLine($"Created Dummy with ID: {result.Id}");

item, _ := persistence.Create(context.Background(), correlationId, dummy1)
fmt.Println("Created Dummy with ID: " + item.Id)

item, _ = persistence.Create(context.Background(), correlationId, dummy2)
fmt.Println("Created Dummy with ID: " + item.Id)

item, _ = persistence.Create(context.Background(), correlationId, dummy3)
fmt.Println("Created Dummy with ID: " + item.Id)

var result = await persistence.create(null, dummy1);
print('Created Dummy with ID: ' + result!.id!);

result = await persistence.create(null, dummy2);
print('Created Dummy with ID: ' + result!.id!);

result = await persistence.create(null, dummy3);
print('Created Dummy with ID: ' + result!.id!);

result = persistence.create(None, dummy1) 
print(f'Created Dummy with ID: {result.id}')
 
result = persistence.create(None, dummy2) 
print(f'Created Dummy with ID: {result.id}')

result = persistence.create(None, dummy3) 
print(f'Created Dummy with ID: {result.id}')
Not available

After creating the persisted objects, we will obtain the following output:

figure 1

As we can see, the memory persistence object allocated a value to the id of dummy3, which we had declared as None.

Read the values from the persistence object

To read the persisted values, we can use the getPageByFilter method that we defined when we created the memory persistence object. Here, we will use a filter to indicate that we are only looking for the dummy2 object.

// get one item
let page = await persistence.getPageByFilter(null, 
    FilterParams.fromTuples('key', 'key 2'), 
    new PagingParams(0, null, true));

console.log('Has ' + page.total + ' items');

for(let item of page.data){
    console.log(`${item.id}, ${item.key}, ${item.content}`);
}


// get one item
var page = await persistence.GetPageByFilterAsync(null,
    FilterParams.FromTuples("key", "key 2"),
    new PagingParams(0, null, true));

Console.WriteLine($"Has {page.Total} items");

foreach (var item in page.Data)
{
    Console.WriteLine($"{item.Id}, {item.Key}, {item.Content}");
}

// get one item
result, _ = persistence.GetPageByFilter(context.Background(), correlationId, cdata.NewFilterParamsFromTuples("key", "key 2"), cdata.NewPagingParams(0, nil, nil))

page, _ = persistence.GetPageByFilter(context.Background(), correlationId, cdata.NewFilterParamsFromTuples("key", "key 2"), cdata.NewPagingParams(0, 0, false))

fmt.Printf("Has %v items \n", page.Total)
for _, v := range page.Data {
	fmt.Printf("%v , %v, %v \n", v.Id, v.Key, v.Content)
}


// get one item
var page = await persistence.getPageByFilter(null,
    FilterParams.fromTuples(['key', 'key 2']), PagingParams(0, null, true));

print('Has ' + page.total.toString() + ' items');

for (var item in page.data) {
  print('${item.id}, ${item.key}, ${item.content}');
}


# get one item 
result = persistence.get_page_by_filter(None, 
                                        FilterParams.from_tuples('key', 'key 2'), 
                                        PagingParams(0, None,True)) 
print(f'Has {result.total} items') 
for item in result.data: 
    print(f'{item.id}, {item.key}, {item.content}')
Not available

The result object is of type DataPage, which has two fields: data and total. The first is a list containing the items on the retrieved page, and the second is the total number of items in our request. After running this code, we will see the following output with the values of the obtained object.

Figure 2

Similarly, we can obtain all the persisted objects by using None as our filter.

See: DataPage

// get all items 
page = await persistence.getPageByFilter(null,
    null,
    new PagingParams(0, null, true));

console.log('Has ' + page.total + ' items');

for (let item of page.data) {
    console.log(`${item.id}, ${item.key}, ${item.content}`);
}

See: DataPage


// get one item
var page = await persistence.GetPageByFilterAsync(null,
        null,
        new PagingParams(0, null, true));

Console.WriteLine($"Has {page.Total} items");

foreach (var item in page.Data)
{
    Console.WriteLine($"{item.Id}, {item.Key}, {item.Content}");
}

See: DataPage

// get all items
page, _ = persistence.GetPageByFilter(correlationId, nil, cdata.NewPagingParams(0, nil, true))

fmt.Printf("Has %v items \n", *page.Total)
for _, v := range page.Data {
	fmt.Printf("%v , %v, %v \n", v.Id, v.Key, v.Content)
}

See: DataPage

// get all items
var page = await persistence.getPageByFilter(
    null, null, PagingParams(0, null, true));
    
print('Has ' + page.total.toString() + ' items');

for (var item in page.data) {
  print('${item.id}, ${item.key}, ${item.content}');
}

See: DataPage

# get all items 
result = persistence.get_page_by_filter(None, 
                                        None, 
                                        PagingParams(0, None, True)) 
print(f'Has {result.total} items') 
for item in result.data: 
    print(f'{item.id}, {item.key}, {item.content}')
Not available

After running the above code, we will obtain the following result:

Figure 3

Update a value in the persistence object

To update a value in the persistence object, we need to use the update method. For example, we can change the content of the dummy2 persisted object to “new content 2”.

result = await persistence.update(null, {id: 'id 1', key: 'key 2', content: 'new content 2'});

result = await persistence.UpdateAsync(null, new Dummy { Id = "id 1", Key = "key 2", Content = "new content 2" });

result, _ = persistence.Update(context.Background(), correlationId, Dummy{"id 1", "key 2", "new content 2"})


result = await persistence.update(null, Dummy(id: 'id 1', key: 'key 2', content: 'new content 2'));

result = persistence.update(None, Dummy('id 1', 'key 2', 'new content 2') )
Not available

To verify the change, we can extract the dummy2 object by applying a filter:

// get all items 
page = await persistence.getPageByFilter(null,
    FilterParams.fromTuples('id', 'id 1'),
    new PagingParams(0, 3));

for (let item of page.data) {
    console.log(`${item.id}, ${item.key}, ${item.content}`);
}

// get all item
var page = await persistence.GetPageByFilterAsync(null,
    FilterParams.FromTuples("id", "id 1"),
    new PagingParams(0, 3, true));

foreach (var item in page.Data)
{
    Console.WriteLine($"{item.Id}, {item.Key}, {item.Content}");
}

// get all items
result, _ = persistence.GetPageByFilter(context.Background(), correlationId, cdata.NewFilterParamsFromTuples("id", "id 1"), cdata.NewPagingParams(0, 3, false))

fmt.Printf("Has %v items \n", page.Total)
for _, v := range page.Data {
	fmt.Printf("%v , %v, %v \n", v.Id, v.Key, v.Content)
}

// get all items
var page = await persistence.getPageByFilter(null, FilterParams.fromTuples(['id', 'id 1']), PagingParams(0, 3));

for (var item in page.data) {
  print('${item.id}, ${item.key}, ${item.content}');
}

# get all items 
result = persistence.get_page_by_filter(None, 
                                        FilterParams.from_tuples('id', 'id 1'), 
                                        PagingParams(0, 3)) 

for item in result.data: 
    print(f'{item.id}, {item.key}, {item.content}')
Not available

And get the updated object:

Figure 4

We can also use the updatePartially function. In this case, we need to specify the id of the object to be updated and a dictionary (map) containing the field to be updated and its new value.

result = await persistence.updatePartially(null, 'id 1', AnyValueMap.fromTuples('content', 'new new content 2'));

var result  = await persistence.UpdatePartially(null, "id 1", AnyValueMap.FromTuples(
    "content", "Partially Updated Content 1"
));

// update patially
updateMap := cdata.NewAnyValueMap(map[string]interface{}{"content": "new new content 2"})
item, _ := persistence.UpdatePartially(context.Background(), correlationId, "id 1", updateMap)

fmt.Printf("%v , %v, %v \n", item.Id, item.Key, item.Content)

result = await persistence.updatePartially(null, 'id 1', AnyValueMap.fromTuples(['content', 'new new content 2']));

result = persistence.update_partially(None, 'id 1', {'content' : 'new new content 2'})
Not available

To verify the change, we can use the filter defined earlier.

// get all items 
page = await persistence.getPageByFilter(null,
    FilterParams.fromTuples('id', 'id 1'),
    new PagingParams(0, null, null));

for (let item of page.data) {
    console.log(`${item.id}, ${item.key}, ${item.content}`);
}

// get one item
var page = await persistence.GetPageByFilterAsync(null,
    FilterParams.FromTuples("id", "id 1"),
    new PagingParams(0, null, true));

foreach (var item in page.Data)
{
    Console.WriteLine($"{item.Id}, {item.Key}, {item.Content}");
}

// get all items
page, _ = persistence.GetPageByFilter(context.Background(), correlationId, cdata.NewFilterParamsFromTuples("id", "id 1"), cdata.NewPagingParams(0, nil, true))

fmt.Printf("get all item\n")
fmt.Printf("Has %v items \n", page.Total)
for _, v := range page.Data {
	fmt.Printf("%v , %v, %v \n", v.Id, v.Key, v.Content)
}

// get all items
var page = await persistence.getPageByFilter(null, FilterParams.fromTuples(['id', 'id 1']), PagingParams(0, null, null));

for (var item in page.data) {
  print('${item.id}, ${item.key}, ${item.content}');
}

# get all items 
result = persistence.get_page_by_filter(None, 
                                        FilterParams.from_tuples('id', 'id 1'), 
                                        PagingParams(0, None, True)) 

for item in result.data: 
    print(f'{item.id}, {item.key}, {item.content}')
Not available

And, we will obtain the updated persisted object.

Figure 5

Delete a value from the persistence object

Similarly, we can delete an object stored in the persistence object by using the deleteById function. In our example, we ask to delete dummy1 by indicating its id.

result = await persistence.deleteById(null, "1");

var result = await persistence.DeleteByIdAsync(null, "1");

result, _ = persistence.DeleteById(context.Background(), correlationId, "1")

result = await persistence.deleteById(null, '1');

result = persistence.delete_by_id(None, "1")
Not available

To verify that the object has been deleted, we can apply a filter and search for it.

// get all items 
page = await persistence.getPageByFilter(null,
    FilterParams.fromTuples('id', '1'),
    new PagingParams(0, null, true));

console.log('Has ' + page.total + ' items');

// get all items 
var page = await persistence.GetPageByFilterAsync(null,
    FilterParams.FromTuples("id", "1"),
    new PagingParams(0, null, true));

Console.WriteLine($"Has {page.Total} items");

// get all item
page, _ = persistence.GetPageByFilter(context.Background(), correlationId, cdata.NewFilterParamsFromTuples("id", "1"), cdata.NewPagingParams(0, nil, true))

fmt.Printf("Has %v items \n", page.Total)


// get all items
var page = await persistence.getPageByFilter(
    null, FilterParams.fromTuples(['id', '1']), PagingParams(0, null, true));

print('Has ' + page.total.toString() + ' items');

# get all items 
result = persistence.get_page_by_filter(None, 
                                        FilterParams.from_tuples('id', '1'), 
                                        PagingParams(0, None, True)) 
print(f'Has {result.total} items')
Not available

As expected, the answer will be:

Figure 6

Wrapping up

In this tutorial, we have seen how to create a memory persistence component and apply CRUD operations to it. Although we used a simple dummy object to create an example, the principles explained continue to apply to more complex objects.