X# STRCONV() implementation proposal

atlopes · Post by **atlopes** » Thu Feb 04, 2021 7:36 pm

The STRCONV function is a notorious miss from the VFP.Net toolkit list of implemented functions.

This is a proposal for its implementation. It tries to accommodate the difference in the nature of VFP and X# (.Net) strings. VFP strings are made of 8 bit ANSI or binary characters, X# strings are Unicode.

The main challenge is to provide the function's nuclear functionality while adding some value to the X# environment.

Some key points

Conversion to Unicode or to DBCS strings return strings.
Base64 and hexadecimal representations of data are treated as strings.
Single-byte strings and UTF-8 are returned as Binary.
Regional identifiers (may) affect single-byte and DBCS strings.
Added two types that represent a Unicode string as a Binary (for LE and BE).

Probably something else will come up with the discussion to follow.

Code: Select all

USING System.Text   
USING System.IO
USING System.Globalization
USING XSharp.Core

FUNCTION Start() AS VOID 

    LOCAL TextNr AS Int
    
    TestNr = 1

    // Base64
    QuickTest(TestNr++, 'StrConv("Abcd", STRCNV_SB_BASE64)', StrConv("Abcd", STRCNV_SB_BASE64), "QWJjZA==")
    QuickTest(TestNr++, 'StrConv("QWJjZA==", STRCNV_BASE64_SB)', StrConv("QWJjZA==", STRCNV_BASE64_SB), (Binary)"Abcd")
    QuickTest(TestNr++, 'StrConv(0h010203fdfeff, STRCNV_SB_BASE64)', StrConv(0h010203fdfeff, STRCNV_SB_BASE64), "AQID/f7/")
    QuickTest(TestNr++, 'StrConv("AQID/f7/", STRCNV_BASE64_SB)', StrConv("AQID/f7/", STRCNV_BASE64_SB), 0h010203fdfeff)

    // Hex
    QuickTest(TestNr++, 'StrConv("Abcd", STRCNV_SB_HEX)', StrConv("Abcd", STRCNV_SB_HEX), "41626364")
    QuickTest(TestNr++, 'StrConv("41626364", STRCNV_HEX_SB)', StrConv("41626364", STRCNV_HEX_SB), (Binary)"Abcd")
    QuickTest(TestNr++, 'StrConv(0h010203fdfeff, STRCNV_SB_HEX)', StrConv(0h010203fdfeff, STRCNV_SB_HEX), "010203FDFEFF")
    QuickTest(TestNr++, 'StrConv("010203FDFEFF", STRCNV_HEX_SB)', StrConv("010203fdfeff", STRCNV_HEX_SB), 0h010203fdfeff)

    // Unicode
    QuickTest(TestNr++, 'StrConv(StrConv(StrConv("Привет мир!", STRCNV_UNI_DBCS, 1251, 1), STRCNV_DBCS_SB), STRCNV_SB_HEX)', ;
        StrConv(StrConv(StrConv("Привет мир!", STRCNV_UNI_DBCS, 1251, 1), STRCNV_DBCS_SB), STRCNV_SB_HEX), "CFF0E8E2E5F220ECE8F021")
    QuickTest(TestNr++, 'StrConv("Привет мир!", STRCNV_UNI_UTF8)', StrConv("Привет мир!", STRCNV_UNI_UTF8), 0hD09FD180D0B8D0B2D0B5D18220D0BCD0B8D18021)
    QuickTest(TestNr++, 'StrConv(e"Belxc3xa9m do Parxc3xa1", STRCNV_UTF8_UNI)', StrConv(e"Belxc3xa9m do Parxc3xa1", STRCNV_UTF8_UNI), "Belém do Pará")
    QuickTest(TestNr++, 'StrConv(0hC8, STRCNV_DBCS_UNI, 1253, 1) + StrConv(0hC8, STRCNV_DBCS_UNI, 1256, 1)', ;
        StrConv(0hC8, STRCNV_DBCS_UNI, 1253, 1) + StrConv(0hC8, STRCNV_DBCS_UNI, 1256, 1), 'Θب')
    // Extras
    QuickTest(TestNr++, 'StrConv("Γειά σου Κόσμε!", STRCNV_UNI_SB)', StrConv("Γειά σου Κόσμε!", STRCNV_UNI_SB), 0h9303B503B903AC032000C303BF03C50320009A03CC03C303BC03B5032100)

    WAIT
    
	RETURN 

FUNCTION QuickTest (Test AS Int, Expression AS String, Result AS USUAL, Expected AS USUAL) AS Void
    
    ? Test, Expression, "->", Result, "(" + IIF(Result == Expected, "Success", "Fail, was expecting " + Expected) + ")"

ENDFUNC

DEFINE STRCNV_SB_DBCS           = 1
DEFINE STRCNV_DBCS_SB           = 2
DEFINE STRCNV_KATA_HIRA         = 3
DEFINE STRCNV_HIRA_KATA         = 4
DEFINE STRCNV_DBCS_UNI          = 5
DEFINE STRCNV_UNI_DBCS          = 6
DEFINE STRCNV_LOWER             = 7
DEFINE STRCNV_UPPER             = 8
DEFINE STRCNV_DBCS_UTF8         = 9
DEFINE STRCNV_UNI_UTF8          = 10
DEFINE STRCNV_UTF8_DBCS         = 11
DEFINE STRCNV_UTF8_UNI          = 12
DEFINE STRCNV_SB_BASE64         = 13
DEFINE STRCNV_BASE64_SB         = 14
DEFINE STRCNV_SB_HEX            = 15
DEFINE STRCNV_HEX_SB            = 16
DEFINE STRCNV_UNI_SB            = 17
DEFINE STRCNV_UNIBE_SB          = 18

DEFINE STRCNV_ID_LCID           = 0
DEFINE STRCNV_ID_CODEPAGE       = 1
DEFINE STRCNV_ID_CHARSET        = 2

FUNCTION StrConv (Expression AS String, ConversionSetting AS Int, RegionalIdentifier = 0 AS Int, RegionalIDType = STRCNV_ID_LCID AS Int) AS USUAL

    RETURN StrConv_Helper(Expression, (Binary)Expression, ConversionSetting, RegionalIdentifier, RegionalIDType)

END FUNCTION

FUNCTION StrConv (Expression AS Binary, ConversionSetting AS Int, RegionalIdentifier = 0 AS Int, RegionalIDType = STRCNV_ID_LCID AS Int) AS USUAL

    RETURN StrConv_Helper((String)Expression, Expression, ConversionSetting, RegionalIdentifier, RegionalIDType)

END FUNCTION

FUNCTION StrConv (Expression AS USUAL, ConversionSetting AS Int, RegionalIdentifier = 0 AS Int, RegionalIDType = STRCNV_ID_LCID AS Int) AS USUAL

    SWITCH VARTYPE(Expression)
        CASE "C"
            LOCAL StrExpression AS String
        
            StrExpression = Expression
            RETURN StrConv(StrExpression, ConversionSetting, RegionalIdentifier, RegionalIDType)

        CASE "Q"
            LOCAL BinExpression AS Binary
        
            BinExpression = Expression
            RETURN StrConv(BinExpression, ConversionSetting, RegionalIdentifier, RegionalIDType)

        OTHERWISE
            THROW ArgumentException{"Incorrect parameter.", nameof(Expression)}
    END SWITCH

END FUNCTION
 
FUNCTION StrConvStr (Expression AS Binary, RegionalIdentifier = 0 AS Int, RegionalIDType = STRCNV_ID_LCID AS Int) AS String

    VAR enc = StrConv_GetEncoding(RegionalIdentifier, RegionalIDType)
    RETURN Encoding.Unicode.GetString(Encoding.Convert(enc, Encoding.Unicode, Expression))
            
END FUNCTION

STATIC FUNCTION StrConv_helper (Expression AS String, BinaryExpression AS Binary, ConversionSetting AS Int, RegionalIdentifier = 0 AS Int, RegionalIDType = STRCNV_ID_LCID AS Int) AS USUAL
    
    LOCAL ReturnStr AS String
    LOCAL ReturnBin AS Binary
    LOCAL ReturnType AS String

    LOCAL defenc = Encoding.Default AS System.Text.Encoding
    LOCAL utf8 = Encoding.UTF8 AS System.Text.Encoding
    LOCAL unicode = Encoding.Unicode AS System.Text.Encoding
    LOCAL enc AS System.Text.Encoding

    LOCAL Indexer AS Int

    SWITCH (ConversionSetting)

        CASE STRCNV_SB_DBCS
             IF RegionalIDType == STRCNV_ID_LCID
                enc = StrConv_GetEncoding(RegionalIdentifier, STRCNV_ID_LCID)
                ReturnStr = defenc.GetString(Encoding.Convert(defenc, enc, BinaryExpression))
                ReturnType = "S"
            ELSE
                THROW ArgumentException {}
            END IF

        CASE STRCNV_DBCS_SB
             IF RegionalIDType == STRCNV_ID_LCID
                enc = StrConv_GetEncoding(RegionalIdentifier, STRCNV_ID_LCID)
                ReturnBin = Encoding.Convert(defenc, enc, BinaryExpression)
                ReturnType = "B"
            ELSE
                THROW ArgumentException {}
            END IF
                
        CASE STRCNV_KATA_HIRA
            ReturnStr = Expression       // not implemented
            ReturnType = "S"

        CASE STRCNV_HIRA_KATA
            ReturnStr = Expression       // not implemented
            ReturnType = "S"

        CASE STRCNV_DBCS_UNI
            enc = StrConv_GetEncoding(RegionalIdentifier, RegionalIDType)
            ReturnStr = unicode.GetString(Encoding.Convert(enc, unicode, BinaryExpression))
            ReturnType = "S"

        CASE STRCNV_UNI_DBCS
            enc = StrConv_GetEncoding(RegionalIdentifier, RegionalIDType)
            ReturnStr = defenc.GetString(Encoding.Convert(unicode, enc, unicode.GetBytes(Expression)))
            ReturnType = "S"

        CASE STRCNV_LOWER
            IF RegionalIDType == STRCNV_ID_LCID
                ReturnBin = defenc.GetBytes(defenc.GetString(BinaryExpression).ToLower(StrConv_GetCulture(RegionalIdentifier)))
                ReturnType = "B"
            ELSE
                THROW ArgumentException {}
            END IF

        CASE STRCNV_UPPER
             IF RegionalIDType == STRCNV_ID_LCID
                ReturnBin = defenc.GetBytes(defenc.GetString(BinaryExpression).ToUpper(StrConv_GetCulture(RegionalIdentifier)))
                ReturnType = "B"
            ELSE
                THROW ArgumentException {}
            END IF

        CASE STRCNV_DBCS_UTF8
            enc = StrConv_GetEncoding(RegionalIdentifier, RegionalIDType)
            ReturnBin = Encoding.Convert(enc, Encoding.UTF8, BinaryExpression)
            ReturnType = "B"

        CASE STRCNV_UNI_UTF8
            ReturnBin = Encoding.Convert(unicode, utf8, unicode.GetBytes(Expression))
            ReturnType = "B"
            
        CASE STRCNV_UTF8_DBCS
            enc = StrConv_GetEncoding(RegionalIdentifier, RegionalIDType)
            ReturnStr = defenc.GetString(Encoding.Convert(Encoding.UTF8, enc, BinaryExpression))
            ReturnType = "S"

        CASE STRCNV_UTF8_UNI
            ReturnStr = unicode.GetString(Encoding.Convert(Encoding.UTF8, Encoding.Unicode, BinaryExpression))
            ReturnType = "S"
            
        CASE STRCNV_SB_BASE64
            ReturnStr = Convert.ToBase64String(BinaryExpression)
            ReturnType = "S"
            
        CASE STRCNV_BASE64_SB
            ReturnBin = Convert.FromBase64String(defenc.GetString(BinaryExpression))
            ReturnType = "B"

        CASE STRCNV_SB_HEX
            ReturnStr = SUBSTR(BinaryExpression.ToString(), 3)
            ReturnType = "S"

        CASE STRCNV_HEX_SB
            VAR SingleBytes = Byte[]{Expression.Length / 2}
            LOCAL Hex AS Int
            Hex = 0
            FOR Indexer = 1 TO SingleBytes.Length
                SingleBytes[Indexer] = Convert.ToByte(Expression.Substring(Hex, 2), 16)
                Hex += 2
            NEXT
            ReturnBin = SingleBytes
            R

robert · Post by **robert** » Thu Feb 04, 2021 8:27 pm

Antonio,
Thanks for the proposal.
I will refrain from comments for now: I have no idea what FoxPro does with this function so I can't judge the implementation.
But what you have done looks very well thought through.
Maybe some small suggestions:
- the default codepage in StrConv_GetEncoding could be XSharp.RuntimeState.WinCodepage (this is read from the OS).
- the argument exceptions could probably include a message
And we will most certainly copy the "QuickTest" code and add it to our unit tests.

Robert

atlopes · Post by **atlopes** » Fri Feb 05, 2021 8:58 am

Robert,

I hope VFP developers will step in, test the implementation, point out its problems, and propose corrections and enhancements.

The "final" version, if we come to that, it's for the X# team to integrate with any adjustments you will find fit. As we discussed, that version will be submitted as a pull request.

Eric Selje · Post by **Eric Selje** » Fri Feb 05, 2021 6:14 pm

I think this is very cool. I'm trying to figure out a way to create tests so that rather than compare against an "expected value" it would compare against what VFP returned so we could ensure they're the same. If we compiled this into a .NET Class we could call that from DotNetBridge in VFP and have the tests in FoxUnit. That'd be the real test whether it's compatible.

StrConv() is an odd function in VFP and mostly obsolete because of the native functionality in .NET, but we're going to need it included if we want our apps to compile and run correctly.

atlopes · Post by **atlopes** » Fri Feb 05, 2021 7:51 pm

Eric,

The expected results in tests come from values returned by VFP, but going through DotNetBridge and FoxUnit will be much more stressful and, thus, clarifying. I'm looking forward to it. And surely it will serve as a foundation for tests of the implementation of other VFP functions.