Author Topic: Memory reclaiming (Read 10902 times)

Spiller · « **on:** November 10, 2009, 08:47:37 pm »

As discussed in the Lynx emulator topic, I'm attaching my code for reclaiming unfreed memory in malloc.h and malloc.c. I'm also sharing my entry_point.c with some alignment warnings fixed and memory reclaiming enabled.

I hope this will benefit other native OS developers. Eventually I will be release my personal SDK which I dubbed 'dingkit', which is aimed at native OS development under Linux without the official SDK.

Kalisiin · « **Reply #1 on:** November 10, 2009, 11:29:05 pm »

Quote from: Spiller on November 10, 2009, 08:47:37 pm

As discussed in the Lynx emulator topic, I'm attaching my code for reclaiming unfreed memory in malloc.h and malloc.c. I'm also sharing my entry_point.c with some alignment warnings fixed and memory reclaiming enabled.

I hope this will benefit other native OS developers. Eventually I will be release my personal SDK which I dubbed 'dingkit', which is aimed at native OS development under Linux without the official SDK.

OK...does this do anything for us, or is it stuff that should be put into the next version update of emu programs?

In other words, if I just put this on my Dingoo, does it help me?

bnolsen · « **Reply #2 on:** November 11, 2009, 03:50:55 am »

Uhh...it's source files to replace other source files that invoke the game's main.
It will require a recompile (header change) and relink to actually include this.

alekmaul · « **Reply #3 on:** November 11, 2009, 06:02:44 am »

Kalisiin, if you're not a developer, the files can't help you, it's only to add to program when developing things (emulators, games, applications, and so on ...).

Spiller, thanks a lot for sharing such info.
Your _malloc_reclaim function can be really usefull !

As you can see in source codes i shared, i'm using a different entry_point without malloc, i give it here again if it can help someone (i'm not sure 256 characters are enough for file/dir length).

Quote

#include <stdlib.h>
#include <string.h>
#include <stdint.h>

extern int main(int, char**);

extern char * __to_locale_ansi(wchar_t*);

char gamePath[256];

void GamePathInit(const char* inPath) {
   uintptr_t i, j;
   for(i = 0, j = 0; inPath != '\0'; i++) {
      if((inPath == '\\') || (inPath == '/'))
         j = i + 1;
   }
   strncpy(gamePath, inPath, j);
}

int GameMain(char* respath) {
   char AppFullName[256];
   char *fullpath;

   strcpy((char *)AppFullName, (char *)respath);
   fullpath = __to_locale_ansi( (wchar_t*)respath );
GamePathInit( fullpath );

   char *userdata[3];
   userdata[0] = AppFullName;
   userdata[1] = gamePath;
   userdata[2] = fullpath;

   int tempOut = main(3, userdata);
   return tempOut;
}

Harteex · « **Reply #4 on:** November 11, 2009, 02:56:20 pm »

Thanks

I will probably modify it to use a linked list structure instead though. This seems like it will leak memory if you do more than _MALLOC_CHECK_MAX allocations.

Kalisiin · « **Reply #5 on:** November 11, 2009, 03:50:05 pm »

Alekmaul...that was sorta what I thought, thanks for clearing that up.

Yeah, I'm not a developer, I'm a user and an admirer. Anyone with the genius to develop this stuff has my admiration!!

I wish I could.

I'd have something of an idea to try to port a version of my own Legend of the Green Dragon on over...but it would require many changes, because it's currently a web-based PHP, multi-player game, and would have to be significantly changed, in order to work out as a one player non-online game.

If anyone was ever willing to try taking it on, I'd be willing to share my source code...including for my original programmed modules, which are not available to the public. I can code PHP, but not this other stuff.

I'd only ask to be an advisor to the creation process of the game if anyone wanted to work on something together. The other person would be basically doing all the coding, because I don't know how to code for Dingoo....or Dingux.

Spiller · « **Reply #6 on:** November 11, 2009, 04:39:34 pm »

Quote from: Harteex on November 11, 2009, 02:56:20 pm

I will probably modify it to use a linked list structure instead though. This seems like it will leak memory if you do more than _MALLOC_CHECK_MAX allocations.

Yes, it will. Please share back if you create an improved version.

Harteex · « **Reply #7 on:** November 12, 2009, 06:56:13 pm »

Yes of course

I'm working on it right now, I'm probably done with it tomorrow.

BTW, _realloc seems like it would leak memory if you send it a NULL region (since malloc is undef'ed).

EDIT: In _realloc you end the function with returning the pointer to region. You should return the return value from realloc instead.

Harteex · « **Reply #8 on:** November 12, 2009, 09:29:22 pm »

Ok, I'm done.
I have not yet tested _realloc though.

Please tell me if you see any mistake. It's very easy to miss some things, that might not show up in the tests either.

I've tested malloc and calloc with the following code:

Code: [Select]

screenWrite("Malloc 1");
 void* v = malloc(1048576); // 1 MB
 screenWrite("Malloc 2");
 void* v2 = malloc(1048576);
 screenWrite("Malloc 3");
 void* v3 = malloc(1048576);
 if (v == NULL || v2 == NULL || v3 == NULL)
 {
  if (v != NULL)
   free(v);
  if (v2 != NULL)
   free(v2);
  if (v3 != NULL)
   free(v3);

  screenWrite("allocation failed, press A");
  waitKey();
  return 1;
 }

 // TEST 1: free memory in the middle of list
 screenWrite("TEST 1");
 free(v2);

 // TEST 2: free memory first in the list
 screenWrite("TEST 2");
 free(v3);

 // TEST 3: don't free v, let the auto reclaim do that

 screenWrite("Done, press A");
 waitKey();

No problems at all on my dingoo running them 35 times each.

If anyone feels like it you could write up a test case on realloc, if not I might do it later...

Anyway, grab the files here: entry_point.c, malloc.c, malloc.h
Or a zip file if you prefer that: memautofree.zip

enjoy

EDIT: Please note that this is NOT threadsafe.
But the first version by Spiller isn't threadsafe either.

Spiller · « **Reply #9 on:** November 13, 2009, 08:26:16 am »

Nice. The reason I chose not to do a linked list is to avoid doing mallocs to prevent leaking mallocs, if you know what I mean.

The problem with your implementation now is that it leaks memory for applications which do not call the reclaim function. My aim was compatibility for both 'old' apps which do not call reclaim yet but manage their mallocs and frees by themselves and 'new' apps which call the reclaim. Also my reclaim function intentionally returns a count of reclaimed regions for debugging purposes. It should return 0 if the app managed its memory properly.

Anyway your implementation is nice as well. I will certainly be merging some bits back to dingkit.

Harteex · « **Reply #10 on:** November 13, 2009, 08:40:03 am »

Quote from: Spiller on November 13, 2009, 08:26:16 am

Nice. The reason I chose not to do a linked list is to avoid doing mallocs to prevent leaking mallocs, if you know what I mean.

The problem with your implementation now is that it leaks memory for applications which do not call the reclaim function. My aim was compatibility for both 'old' apps which do not call reclaim yet but manage their mallocs and frees by themselves and 'new' apps which call the reclaim. Also my reclaim function intentionally returns a count of reclaimed regions for debugging purposes. It should return 0 if the app managed its memory properly.

Anyway your implementation is nice as well. I will certainly be merging some bits back to dingkit.

If you do free for all memory you've allocated you will not need to call the reclaim function on my code either, as with your implementation. Unless of course I have a bug somewhere..

But otherwise, where would I leak memory in the case you do all free yourself?

EDIT: Clarification: If you do free correctly, _malloc_list_root would be NULL, and no memory should be allocated.

Spiller · « **Reply #11 on:** November 13, 2009, 09:19:13 am »

You're right and I was wrong. Sorry for the confusion.

Spiller · « **Reply #12 on:** November 15, 2009, 11:56:58 am »

Hi Harteex, just had another look at it. Why are you wrapping calloc? The original calloc implementation is calling malloc so should be covered already by wrapping malloc with _malloc... Am I missing something again?

Harteex · « **Reply #13 on:** November 15, 2009, 01:31:12 pm »

Quote from: Spiller on November 15, 2009, 11:56:58 am

Hi Harteex, just had another look at it. Why are you wrapping calloc? The original calloc implementation is calling malloc so should be covered already by wrapping malloc with _malloc... Am I missing something again?

Ah.. well that's a mistake on my part

Harteex · « **Reply #14 on:** November 15, 2009, 07:05:15 pm »

Ok, I have updated the files for this problem. I think it should cause problems, but during my testing I didn't have any. Very odd.
Anyway, thanks Spiller for noticing!

Files are updated in place:
entry_point.c, malloc.c, malloc.h
Or a zip file if you prefer that: memautofree.zip

flatmush · « **Reply #15 on:** November 30, 2009, 05:04:39 pm »

I've looked at both of your implementations and they are very good, I'll try to optimize harteex's version and put it in the lib.

What I'm going to do is:
- Use linked hash tables to lower the overhead when reallocating/freeing and so there isn't double the number of allocations.
- Use the new allocation structures to correctly implement memalign.

Gotta remember that linked lists are slow in hardware, and malloc/free are called lots especially in cpp code.

It would be useful if someone who has more time to play with could find out where the malloc list is stored so that we wouldn't have to store our own wrapper copy of it.

Dingoonity.org

Author Topic: Memory reclaiming (Read 10902 times)

Spiller (OP)

Memory reclaiming

Kalisiin

Re: Memory reclaiming

bnolsen

Re: Memory reclaiming

alekmaul

Re: Memory reclaiming

Harteex

Re: Memory reclaiming

Kalisiin

Re: Memory reclaiming

Spiller (OP)

Re: Memory reclaiming

Harteex

Re: Memory reclaiming

Harteex

Re: Memory reclaiming

Spiller (OP)

Re: Memory reclaiming

Harteex

Re: Memory reclaiming

Spiller (OP)

Re: Memory reclaiming

Spiller (OP)

Re: Memory reclaiming

Harteex

Re: Memory reclaiming

Harteex

Re: Memory reclaiming

flatmush

Re: Memory reclaiming