MiniRobotLanguage (MRL)
Collections / Fast-Arrays
Collections are sets of unique Items * Arrays are the same just not unique
Intention
The SPR provides 2049 memory areas internally (0 - 2048).
These can be used either as an array or as a collection, or even as both in parallel.
The memory area is specified as the first parameter in these commands as a number from 0 to 2048.
Let's assume P1 is 0. This means we are using memory area 0.
Depending on the command used, this memory area will then be used as a collection or as an array - or as both at he same time!
What Collections / Arrays are available?
Currently the Smart Package Robot supports 2 Sorts of Collections. These are numbered from 0 to 2048.
Collection Type 1: "Word-Collections".
They are numbered from 0 to 1024.
Collection Type 2: "Checksum-Collections".
They are numbered from 1025 to 2048.
The difference between collections and arrays lies in the way they are addressed and stored.
In arrays, you can access any element at any position and, for example, sort the elements.
In collections, you add or remove elements and typically do not concern yourself with where these elements are stored in the collection.
Collections usually do not need to be sorted.
Additionally, there is the important difference that each element in a collection is unique. This means you cannot add the same element to a collection twice; the commands prevent this.
This is unlike arrays, where you can add the same element multiple times.
Note that each memory area, whether 0, 1 to 1024, represents an area that can be used for collections or arrays and can be used simultaneously.
This means I can also create a collection in this memory area using collection commands and later use this collection like an array.
Conversely, I can take an array and use it like a collection, which would then also have duplicate elements.
When I create an array and then use the memory area with collection commands, it does not check for duplicate elements; this is only done when adding new elements.
So, importantly, there are 0 to 2048 memory areas, and these can be used simultaneously as arrays and collections.
You can use them like Collections using ATC. or like Arrays using ATA.
' This script demonstrates how to utilize collections to clean up a text file filled with thousands of links,
' ensuring that each link appears only once in the resulting file.
' By leveraging collections, we guarantee the uniqueness of elements, eliminating any duplicates.
' This method is the fastest way to achieve this, with the script executing in just a second.
' #DataCleaning #UniqueLinks #PythonScript #Collections #Efficiency
$$FIN=?exeloc\SV_Links.txt
$$FIL=?exeloc\Uniqe_Links.txt
DEL.$$FIL
$$CNT=0
TXT.Load Text File|$$FIN|$$LIN
DBP.Got $$LIN Lines
FOR.$$LTX|0|$$LIN
TXT.Get Line Trimmed|$$LTX|$$LTT
ATC.0|$$LTT
NEX.
FEC.Append|$$LTK|0
MBX.Done
ENR.
:Append
JNS.$$LTK|Over
$$TXT=$$LTK$crlf$
ATF.$$FIL|$$TXT
:Over
RET.
What is the difference between these two types?
In Type 1 "Word-Collections", the Strings are directly stored.
Therefore Type 1 Collections can handle Binary Data in any length (32-bit limits). They store the complete "Word" and therefore may use a lot Memory for large chunks of data.
In Type 2 "Checksum-Collections", the Strings are processed to a 20 Byte Checksum and only the 20-Byte Checksum is stored without any delimiter.
Therefore Type 2 can be used for binary data and for larger strings up to hundreds of MB. And will still only result in a 20 Byte Checksum (8-Byte lengh + 16 Bye Checksum) per Item.
As a Checksum can not be guarantee to deliver perfect results under all circumstances, there is a theoretical weakness to not add practically different elements that may have the same checksum and lenght.
Checksums are designed to verify the integrity of data or detect errors in data transmission. However, they do not offer a perfect solution under all circumstances due to inherent limitations. One theoretical weakness of relying solely on checksums for identifying unique elements is the possibility of a collision. A collision occurs when two distinct data elements produce the same checksum value. Although the likelihood of such an event might be low, especially with sophisticated checksum algorithms, it is not zero. This means that there's a chance that two practically different elements could be considered the same due to having identical checksums. Consequently, while checksums are a valuable tool for error detection and data verification, they cannot absolutely guarantee the uniqueness of every distinct element based solely on their checksum values. This limitation underscores the importance of employing additional methods or algorithms to mitigate the risk of collisions, thereby ensuring more reliable differentiation between unique data elements.
Which Collections should I use?
For normal use you will generally use Type 1 Collections, numbered from 0 to 1024.
Use Type 2 only if you have large strings in a large amount to handle.
How do you decide which Collections you use?
If the Collection number is 0 to 1024 this will use Collections of Type 1.
If the Collection number is 1025 to 2048 the used Collections will be Type 2.
I can use the Collections also like String-Arrays?
You can use the Commands ATA. (Add to ARRAY) instead of ATC. (Add to Collection).
Using the Array Commands there is no check if an Element is unique (double?) or not.
You can use CGE. (Collection Get Element) and CSE.(Collection Set Element) to access every single element in a Collection / Array.
Is there any sort of ReDim needed for the String-Arrays?
No, accessing a Element in an String Array (read or write) will automatically Redim the size of that Array to this size.
The Array will keep this size unless you use the RFC. or CLC.- Command.
What can i do with Collections?
•You can add "Elements" to these Collections.
•You can test if an Element is already in a Collection.
•You can remove Elements from Collections and you can
•Clear Collections.
•Collections are always "add if unique". They will never allow to add the same element twice.
•There are 2 Jump Commands JIU. and JNU. to tst if a Element is already inside a Collection.
FOR.$$LC3|1|5
TST.Generate PreName|$$NAM
CGC.0|$$NUA
DBP.Bevore: $$NUA adding: $$NAM
ATC.0|$$NAM
CGC.0|$$NUM
DBP.After: $$NUM
IVV.$$NUA=$$NUM
DBP.$$NAM not added (is already in Collection).
EIF.
NEX.
ENR.
Collections are groups of character strings that are separated by a null byte. Therefore Elements must never Contain a CHR(0) (Null-Byte).
Collections always contain unique strings.
This means that if you want to add the same element to a collection twice, it will not be added a second time.
Therefore, collections are groups of unique elements.
Collections can be used to test whether you have already processed an element, string or number.
This is because you can test whether an element already exists in a collection or not.
And that's what collections are good for.
They help you to keep your data processing unique.
Collections cannot contain binary data because the data in collections is separated by null characters.
You can transfer collections into variables and restore collections from variables.
Collections are always global, they are not limited to module boundaries.
This means even if you switch to another module, you have always the same collections. Collections do not recognize module boundaries, they are global.
What do those Jumps do?
There are 2 Jumps, these are:
JIU. - Jump if unique
This jump will jump to P3, if the given P2 is not yet in the Collection P1
JNU. - Jump if not unique
This jump will jump to P3, if the given P2 is already in the Collection P1
Examples
'#######################################################################
' Remove double Mailadresses from a list with Mailadresses in a Textfile
' separated with $crlf$
'#######################################################################
'#EXE:?path\
'#SPI:ForceWrite
$$TMA=#dtime#
IEM.
VAR.$$FIA=$cmdexe$
ELS.
VAR.$$FIA=?exeloc\active.txt
EIF.
VAR.$$OUF=?exeloc\Result_#date#.txt
NEF.$$FIA
$$ERR=Die Quelldatei: $$FIA$crlf$existiert nicht.
GTO.ero
EIF.
'-----------------------------------------------------------
TXT.Load_Textfile|$$FIA|$$LIC
' Convert $LF to $crlf$ if needed.
'TXT.lfp
TXT.Get_Line_Count|$$LIC
IVV.$$LIC<5
$$ERR=File is empty?
GTO.ero
EIF.
PRT.$$LIC Lines loaded.
'++++++++++++++++++++++++++++++++
'++++++++++++++++++++++++++++++++
' Output Variable
$$COL=
FOR.$$LUA|1|$$LIC
TXT.Get_Line_Trimmed|$$LUA|$$LIN
BCH.$$LIN|;|1|$$LIN
VBT.$$LIN|All
JNS.$$LIN|remo
' prepare Lower case for collection
VTL.$$LIN|$$TST
' Test if is in Collection already, if so jump to next line
JNU.0|$$TST|Iter
ATC.0|$$TST
' Prepare Line to add to result
$$ADD=$$LIN$crlf$
' Add Original-Mailadress to result
VBA.$$COL|$$ADD
PRT. Added: $$LIN
GTO.Iter
:remo
PRT.Skipped:$$TST
:Iter
NEX.
SRT.$$COL|$crlf$|a
CTF.$$OUF|$$COL
'++++++++++++++++++++++++++++++++
'++++++++++++++++++++++++++++++++
' These lines process and save the result.
TXT.Get_Line_Count|$$LID
GSB.Calc
DBP.$$TXA
DBP.$$LID Lines left.
'TXT.lfr
'TXT.Save_File|$$OUF
'++++++++++++++++++++++++++++++++
'++++++++++++++++++++++++++++++++
:enx
GSB.Calc
IVV.$$TMA>1000
CAL.$$TMA=$$TMA/1000
$$TXT=Programm wird beendet nach $$TMA Sekunden.$$TXA
ELS.
$$TXT=Programm wird beendet nach $$TMA ms.$$TXA
EIF.
MBX.!
PRT.Starting Excel.
EXO.$$OUF
MBT.$$TXT|Programm Ende|9|32
ENR.
'-----------------------------------------------------------
:Calc
CAL.$$TMA=#dsince#*1000
CAL.$$TMB=$$TMA/100|i
CAL.$$TMC=$$TMB/10
CAL.$$LIF=($$LIC*1000)/$$TMA|i
$$TXA=$crlf$Processing $$LIF Lines per Second.
PRT.$$LID Lines processed in $$TMC Seconds.
RET.
'-----------------------------------------------------------
:ero
PRT.$$ERR
GTO.enx
'#######################################################################
'************************************************************
' Complex Script for Dynamic Collection Management
'************************************************************
' Initialize collection 0 and ensure it's empty
VAR.$$C00=0
CLC.$$C00
DBP.Initializing and clearing collection 0.
' Add several elements to the collection
VAR.$$IT1=Apple
ATC.$$C00|$$IT1
VAR.$$IT2=Banana
ATC.$$C00|$$IT2
VAR.$$IT3=Cherry
ATC.$$C00|$$IT3
DBP.Apple, Banana, and Cherry added to collection 0.
' Check if Banana is in the collection using IIC.
IIC.$$C00|$$IT2
DBP.Banana is confirmed to be in the collection.
ELS.
DBP.Banana is not in the collection, which is unexpected.
EIF.
' Remove Banana from the collection
RFC.$$C00|$$IT2
DBP.Banana removed from collection 0.
' Attempt to add Banana again, but first check if it's not in the collection using NIC.
NIC.$$C00|$$IT2
DBP.Banana is not in the collection, proceeding to add.
ATC.$$C00|$$IT2
ELS.
DBP.Banana is unexpectedly found in the collection, not added.
EIF.
' Backup the collection to a variable
CTV.$$C00|$$DAT
DBP.Collection 0 backed up to $$DAT.
' Clear and restore the collection to verify the backup and restore process
CLC.$$C00
DBP.Collection 0 cleared.
CFV.$$C00|$$DAT
DBP.Collection 0 restored from $$DAT.
' Verify the restoration by checking the count of elements
CGC.$$C00|$$CNT
DBP.Collection 0 now contains $$CNT elements post-restoration.
' Verify specific elements' presence post-restoration
' Check for Apple
IIC.$$C00|$$IT1
DBP.Apple is confirmed to be in the collection post-restoration.
ELS.
DBP.Apple is not in the collection post-restoration, which is unexpected.
EIF.
' Check for Cherry
IIC.$$C00|$$IT3
DBP.Cherry is confirmed to be in the collection post-restoration.
ELS.
DBP.Cherry is not in the collection post-restoration, which is unexpected.
EIF.
DBP.Script concluded. Dynamic management of collection 0 demonstrated.
' End of script
ENR.
Script Explanation:
Initialization: The script starts by initializing and clearing collection 0.
Addition: Adds "Apple", "Banana", and "Cherry" to the collection, demonstrating the ATC. command.
Presence Check: Uses IIC. to confirm "Banana" is in the collection before removing it.
Removal and Re-Addition: Removes "Banana" with RFC., then checks its absence with NIC. before adding it back, showcasing conditional element management.
Backup and Restore: The collection's content is backed up to a variable with CTV., then the collection is cleared and restored with CFV., illustrating data preservation techniques.
Verification: Finally, CGC. is used to verify the count of elements in the restored collection, with additional checks using IIC. to confirm the presence of specific elements post-restoration.
'***********************************
' CLC.-Sample
' Collection Management Sample
'***********************************
' Set the identifier for the collection to be used
VAR.$$CNU=1025
' Generate a prename and assign it to variable $$NAA
TST.Generate PreName|$$NAA
' Get the count of elements in the collection before any additions
CGC.$$CNU|$$NUA
' Output the current count in the collection
DBP.Before: Collection $$CNU has $$NUA elements.
' Add the generated prename to collection $$CNU
ATC.$$CNU|$$NAA
' Get and output the count of elements after attempting to add the prename
CGC.$$CNU|$$NUA
DBP.After: Collection $$CNU has $$NUA elements.
' Attempt to add the same prename to collection $$CNU two more times
ATC.$$CNU|$$NAA
ATC.$$CNU|$$NAA
' Get and output the count of elements to verify that duplicates were not added
CGC.$$CNU|$$NUA
DBP.No Change Expected: Collection $$CNU still has $$NUA elements.
' Generate another prename and assign it to variable $$NAB
TST.Generate PreName|$$NAB
' Output the current count of elements before adding a new prename
CGC.$$CNU|$$NUA
DBP.Before: Collection $$CNU has $$NUA elements. Attempting to add: $$NAB
' Add the new prename to collection $$CNU
ATC.$$CNU|$$NAB
' Get and output the updated count of elements after adding the new prename
CGC.$$CNU|$$NUA
DBP.After Adding: Collection $$CNU now has $$NUA elements.
' Remove the first generated prename from collection $$CNU
RFC.$$CNU|$$NAA
' Get and output the count of elements after removal
CGC.$$CNU|$$NUA
DBP.After Removal: Collection $$CNU has $$NUA elements remaining.
' Alert box indicating the end of processing
MBX.!
' End of script
ENR.
'***********************************
' MRL Collection Management Sample with Jump Commands
'***********************************
' Set the identifier for the collection to be used
VAR.$$CNU=1025
' Generate a prename and assign it to variable $$NAA
TST.Generate PreName|$$NAA
' Check if the prename $$NAA is unique in the collection $$CNU
JIU.$$CNU|$$NAA|name_is_unique
DBP.$$NAA is not unique and won't be added.
JMP.end_of_script
:name_is_unique
' Since $$NAA is unique, it will be added to the collection $$CNU
ATC.$$CNU|$$NAA
:end_of_script
' Get the count of elements in the collection
CGC.$$CNU|$$NUA
' Output the current count in the collection
DBP.Collection $$CNU has $$NUA elements.
' Generate another prename and assign it to variable $$NAB
TST.Generate PreName|$$NAB
' Check if the prename $$NAB is not unique in the collection $$CNU
JNU.$$CNU|$$NAB|name_not_unique
DBP.$$NAB is unique and will be added.
' Since $$NAB is unique, it will be added to the collection $$CNU
ATC.$$CNU|$$NAB
JMP.post_addition
:name_not_unique
DBP.$$NAB is not unique and won't be added again.
:post_addition
' Get and output the updated count of elements after attempting to add new prenames
CGC.$$CNU|$$NUA
DBP.Collection $$CNU now has $$NUA elements.
' End of script
ENR.
'************************************************************
' Advanced Collection Management Testing Script
'************************************************************
' Set the identifier for the collection to be used
VAR.$$CNU=1027
' Start a loop to simulate generating and processing multiple prenames
FOR.$$LC3|1|10
' Generate a prename and assign it to variable $$NAM
TST.Generate PreName|$$NAM
' Log the generated prename
DBP.Append Generated prename: $$NAM
' Check if the prename $$NAM is unique in the collection $$CNU
JIU.$$CNU|$$NAM|name_is_unique
' If the prename is not unique, log the event and skip adding
DBP.Append $$NAM is already in Collection $$CNU, skipping.
JMP.continue_loop
:name_is_unique
' If the prename is unique, add it to the collection
ATC.$$CNU|$$NAM
' Log the successful addition
DBP.Append Added $$NAM to Collection $$CNU.
:continue_loop
' Get and output the updated count of elements in the collection
CGC.$$CNU|$$NUA
DBP.Current size of Collection $$CNU: $$NUA elements.
' Generate a decision point to simulate various collection manipulations
RND.1|3|$$DEC
IVV.$$DEC=1
' Case 1: Try to re-add the same prename to check for duplicates
ATC.$$CNU|$$NAM
DBP.Append Re-attempted to add $$NAM to Collection $$CNU.
EIF.
IVV.$$DEC=2
' Case 2: Remove the prename from the collection
RFC.$$CNU|$$NAM
DBP.Append Removed $$NAM from Collection $$CNU.
EIF.
IVV.$$DEC=3
' Case 3: Check if the prename was actually removed
JNU.$$CNU|$$NAM|name_not_unique
DBP.Append Verified that $$NAM is no longer in Collection $$CNU.
:name_not_unique
DBP.Append $$NAM is still in Collection $$CNU.
EIF.
' End of current loop iteration
NEX.
' End of loop - clean up the collection
CLC.$$CNU
DBP.Collection $$CNU has been cleared.
MBX.!
ENR.
'************************************************************
' Enhanced MRL Collection Command Testing Script
'************************************************************
' PART 1: Testing with Collection Number 0
DBP.Testing collection number 0.
' Initialize the collection identifier to collection 0
VAR.$$C00=0
' Ensure the collection is empty before starting
CLC.$$C00
DBP.Collection 0 cleared.
' Add unique names to the collection
VAR.$$N10=John
ATC.$$C00|$$N10
VAR.$$N20=Jane
ATC.$$C00|$$N20
DBP.Names John and Jane added to collection 0.
' Attempt to add a duplicate name
VAR.$$D10=John
ATC.$$C00|$$D10
DBP.Attempted to add duplicate name John to collection 0.
' Check the count of names in the collection
CGC.$$C00|$$CT0
DBP.Collection 0 has $$CT0 elements. Expected: 2 (if ATC. works correctly).
' Remove a name from the collection
RFC.$$C00|$$N20
DBP.Name Jane removed from collection 0.
' Check the final count
CGC.$$C00|$$CT0
DBP.Final count in collection 0: $$CT0 elements. Expected: 1 (if RFC. works correctly).
' PART 2: Testing with Collection Number 1027
DBP.Testing collection number 1027.
' Initialize the collection identifier to collection 1027
' Note: This collection number might be beyond the valid range, serving purely for demonstration.
VAR.$$C27=1027
' Clear the collection, assuming the system can handle this index for educational purposes
CLC.$$C27
DBP.Collection 1027 cleared. (Note: This step is theoretical if 1027 is outside valid range.)
' Add unique names to the collection
VAR.$$N17=Alice
ATC.$$C27|$$N17
VAR.$$N27=Bob
ATC.$$C27|$$N27
DBP.Names Alice and Bob added to collection 1027.
' Attempt to add a duplicate name
VAR.$$D17=Alice
ATC.$$C27|$$D17
DBP.Attempted to add duplicate name Alice to collection 1027.
' Check the count of names in the collection
CGC.$$C27|$$CT7
DBP.Collection 1027 has $$CT7 elements. Expected: 2 (if ATC. works correctly and 1027 is valid).
' Remove a name from the collection
RFC.$$C27|$$N17
DBP.Name Alice removed from collection 1027.
' Check the final count
CGC.$$C27|$$CT7
DBP.Final count in collection 1027: $$CT7 elements. Expected: 1 (if RFC. works correctly and 1027 is valid).
' End of script
ENR.
'--- Testing with Collection 0 ---
DBP.Testing with collection number 0.
' Initialize collection 0 and ensure it's empty
VAR.$$C00=0
CLC.$$C00
DBP.Collection 0 cleared.
' Add a unique name to the collection
VAR.$$N10=John
ATC.$$C00|$$N10
DBP.John added to collection 0.
' Test uniqueness before attempting to add John again
JIU.$$C00|$$N10|john_is_unique
' John is already in the collection
DBP.John is considered not unique.
JMP.continue_test
:john_is_unique
' This should not execute because John is already in the collection
DBP.John is unexpectedly considered not in the collection.
:continue_test
' Attempt to add John again to test ATC.'s prevention of duplicates
ATC.$$C00|$$N10
' Check the count to confirm John wasn't added again
CGC.$$C00|$$CT0
IVV.$$CT0=1
DBP.Correct, John was not added again, collection has $$CT0 names.
ELS.
DBP.Error: Duplicate John was added.
EIF.
' Add another unique name, Jane, to the collection
VAR.$$N20=Jane
ATC.$$C00|$$N20
DBP.Jane added to collection 0.
'--- Testing with Collection 1027 ---
DBP.Testing with collection number 1027 (beyond standard range).
' Assuming collections beyond 1024 are handled, initialize collection 1027 and clear it
VAR.$$C27=1027
CLC.$$C27
DBP.Collection 1027 cleared. (Note: Assumes handling beyond standard range.)
' Add a name to collection 1027
VAR.$$N27=Alice
ATC.$$C27|$$N27
DBP.Alice added to collection 1027.
' Test uniqueness with JNU. before removing Alice
JNU.$$C27|$$N27|alice_not_unique
' This part should be skipped since Alice was initially added
DBP.Alice unexpectedly considered not in the collection.
:alice_not_unique
' Since Alice is in the collection, this should execute
DBP.Alice is confirmed in the collection.
' Remove Alice to test RFC.
RFC.$$C27|$$N27
DBP.Alice removed from collection 1027.
' Finalize script with confirmation message
MBX.Collection command tests completed.
' End of script
ENR.
Speed-Considerations:
Type 1 - Collections, Speed-Dump inn Free-Run Mode.
Type 2 - Collections, Speed-Dump inn Free-Run Mode.
Type 1 Collections are a bit faster then Type-2 Collections on smaller Strings.
Directly using a Value like: RFC.9|$$STR may also add another up to 20% Speed as there is no variable to resolve.
As a rule of thumb you can make up to 100.000 Collection Operations in a Second.
Remarks
-
Limitations:
Currently the Smart Package Robot supports 1025 Collections that are numbered from 0 to 1024 and Type 2 -"Checksum"-Collections from 1025 to 2048.
Type 1 Collections are groups of character strings that are separated by a null byte. Therefore Elements must never Contain a CHR(0) (Null-Byte).
Type 2 Collections will contain a 16 Byte Checksum of an Element and do not have any delimiter.
See also:
•